next

Edit

Disk Fill Experiment Details

Experiment Metadata

Type Description Tested K8s Platform
Chaos Fill up Ephemeral Storage of a Pod GKE, EKS, AKS

Prerequisites

  • Ensure that Kubernetes Version > 1.16
  • Ensure that the Litmus Chaos Operator is running by executing kubectl get pods in operator namespace (typically, litmus). If not, install from here
  • Ensure that the disk-fill experiment resource is available in the cluster by executing kubectl get chaosexperiments in the desired namespace If not, install from here
  • Cluster must run docker container runtime
  • Either an appropriate Ephemeral Storage Requests and Limits should be set for the application or EPHEMERAL_STORAGE_MEBIBYTES ENV should be defined in the ChaosEngine before running the experiment. An example specification is shown below:
apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  containers:
  - name: db
    image: mysql
    env:
    - name: MYSQL_ROOT_PASSWORD
      value: "password"
    resources:
      requests:
        ephemeral-storage: "2Gi"
      limits:
        ephemeral-storage: "4Gi"
  - name: wp
    image: wordpress
    resources:
      requests:
        ephemeral-storage: "2Gi"
      limits:
        ephemeral-storage: "4Gi"

Entry-Criteria

  • Application pods are healthy before chaos injection.

Exit-Criteria

  • Application pods are healthy post chaos injection.

Details

  • Causes Disk Stress by filling up the ephemeral storage of the pod on any given node.
  • Causes the application pod to get evicted if the capacity filled exceeds the pod's ephemeral storage limit.
  • Tests the Ephemeral Storage Limits, to ensure those parameters are sufficient.
  • Tests the application's resiliency to disk stress/replica evictions.

Integrations

  • Disk Fill can be effected using the chaos library: litmus, which makes use of dd to create a file of specified capacity on the node.
  • The desired chaoslib can be selected by setting the above options as value for the env variable LIB

Steps to Execute the Chaos Experiment

  • This Chaos Experiment can be triggered by creating a ChaosEngine resource on the cluster. To understand the values to provide in a ChaosEngine specification, refer Getting Started

  • Follow the steps in the sections below to create the chaosServiceAccount, prepare the ChaosEngine & execute the experiment.

Prepare chaosServiceAccount

  • Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment.

Sample Rbac Manifest

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: disk-fill-sa
  namespace: default
  labels:
    name: disk-fill-sa
    app.kubernetes.io/part-of: litmus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: disk-fill-sa
  namespace: default
  labels:
    name: disk-fill-sa
    app.kubernetes.io/part-of: litmus
rules:
- apiGroups: [""]
  resources: ["pods","events"]
  verbs: ["create","list","get","patch","update","delete","deletecollection"]
- apiGroups: [""]
  resources: ["pods/exec","pods/log","replicationcontrollers"]
  verbs: ["list","get","create"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["create","list","get","delete","deletecollection"]
- apiGroups: ["apps"]
  resources: ["deployments","statefulsets","daemonsets","replicasets"]
  verbs: ["list","get"]
- apiGroups: ["apps.openshift.io"]
  resources: ["deploymentconfigs"]
  verbs: ["list","get"]
- apiGroups: ["argoproj.io"]
  resources: ["rollouts"]
  verbs: ["list","get"]
- apiGroups: ["litmuschaos.io"]
  resources: ["chaosengines","chaosexperiments","chaosresults"]
  verbs: ["create","list","get","patch","update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: disk-fill-sa
  namespace: default
  labels:
    name: disk-fill-sa
    app.kubernetes.io/part-of: litmus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: disk-fill-sa
subjects:
- kind: ServiceAccount
  name: disk-fill-sa
  namespace: default

Note: In case of restricted systems/setup, create a PodSecurityPolicy(psp) with the required permissions. The chaosServiceAccount can subscribe to work around the respective limitations. An example of a standard psp that can be used for litmus chaos experiments can be found here.

Prepare ChaosEngine

  • Provide the application info in spec.appinfo
  • Provide the auxiliary applications info (ns & labels) in spec.auxiliaryAppInfo
  • Override the experiment tunables if desired in experiments.spec.components.env
  • To understand the values to provided in a ChaosEngine specification, refer ChaosEngine Concepts

Supported Experiment Tunables

Variables Description Specify In ChaosEngine Notes
FILL_PERCENTAGE Percentage to fill the Ephemeral storage limit Mandatory Can be set to more than 100 also, to force evict the pod
TARGET_CONTAINER Name of container which is subjected to disk-fill Optional If not provided, the first container in the targeted pod will be subject to chaos
CONTAINER_PATH Storage Location of containers Optional Defaults to '/var/lib/docker/containers'
TOTAL_CHAOS_DURATION The time duration for chaos insertion (sec) Optional Defaults to 60s
TARGET_PODS Comma separated list of application pod name subjected to disk fill chaos Optional If not provided, it will select target pods randomly based on provided appLabels
DATA_BLOCK_SIZE It contains data block size used to fill the disk(in KB) Optional Defaults to 256, it supports unit as KB only
PODS_AFFECTED_PERC The Percentage of total pods to target Optional Defaults to 0 (corresponds to 1 replica), provide numeric value only
LIB The chaos lib used to inject the chaos Optional Defaults to `litmus` supported litmus only
LIB_IMAGE The image used to fill the disk Optional Defaults to `litmuschaos/go-runner:latest`
RAMP_TIME Period to wait before injection of chaos in sec Optional
SEQUENCE It defines sequence of chaos execution for multiple target pods Optional Default value: parallel. Supported: serial, parallel
EPHEMERAL_STORAGE_MEBIBYTES Ephemeral storage which need to fill (unit: MiBi) Optional
INSTANCE_ID A user-defined string that holds metadata/info about current run/instance of chaos. Ex: 04-05-2020-9-00. This string is appended as suffix in the chaosresult CR name. Optional Ensure that the overall length of the chaosresult CR is still < 64 characters

Sample ChaosEngine Manifest

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: nginx-chaos
  namespace: default
spec:
  # It can be active/stop
  engineState: 'active'
  #ex. values: ns1:name=percona,ns2:run=nginx  
  auxiliaryAppInfo: ''
  appinfo:
    appns: 'default'
    applabel: 'app=nginx'
    appkind: 'deployment'
  chaosServiceAccount: disk-fill-sa
  experiments:
    - name: disk-fill
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '60' 

            # specify the fill percentage according to the disk pressure required
            - name: FILL_PERCENTAGE
              value: '80'

            - name: PODS_AFFECTED_PERC
              value: ''

            # Provide the container runtime path
            # Default set to docker container path
            - name: CONTAINER_PATH
              value: '/var/lib/docker/containers'

Create the ChaosEngine Resource

  • Create the ChaosEngine manifest prepared in the previous step to trigger the Chaos.

    kubectl apply -f chaosengine.yml

  • If the chaos experiment is not executed, refer to the troubleshooting section to identify the root cause and fix the issues.

Watch Chaos progress

  • View the status of the pods as they are subjected to disk stress.

    watch -n 1 kubectl get pods -n <application-namespace>

  • Monitor the capacity filled up on the host filesystem

    watch -n 1 du -kh /var/lib/docker/containers/<container-id>

Abort/Restart the Chaos Experiment

  • To stop the pod-delete experiment immediately, either delete the ChaosEngine resource or execute the following command:

    kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"stop"}}'

  • To restart the experiment, either re-apply the ChaosEngine YAML or execute the following command:

    kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"active"}}'

Notes:

  • The abort will stop further fill of the local disk, but doesn't reclaim the used capacity. This is a manual operation as of today. The auto-rollback, i.e., in this case reclaim of currently filled disk-space will be implemented in a future release

  • However, upon graceful completion of the experiment (i.e.,un-aborted), the space is automatically reclaimed as the chaos impact is reverted.

Check Chaos Experiment Result

  • Check whether the application is resilient to the container kill, once the experiment (job) is completed. The ChaosResult resource name is derived like this: <ChaosEngine-Name>-<ChaosExperiment-Name>.

    kubectl describe chaosresult nginx-chaos-disk-fill -n <application-namespace>

Disk Fill Experiment Demo

  • A sample recording of this experiment execution is provided here.
← Pod DNS SpoofNode CPU Hog →