1.7.0

Edit

Container Kill Experiment Details

Experiment Metadata

Type Description Tested K8s Platform
Generic Kill one container in the application pod GKE, Packet(Kubeadm), Minikube, EKS, AKS

Prerequisites

  • Ensure that the Litmus Chaos Operator is running by executing kubectl get pods in operator namespace (typically, litmus). If not, install from here
  • Ensure that the container-kill experiment resource is available in the cluster by executing kubectl get chaosexperiments in the desired namespace. If not, install from here
  • Cluster should have docker runtime, if the chaos_lib is pumba. It should have containerd runtime in cluster, if the chaos_lib is litmus.

Entry Criteria

  • Application pods are healthy before chaos injection

Exit Criteria

  • Application pods are healthy post chaos injection

Details

  • Pumba chaoslib details
    • It can kill the multiple containers (can be tuned by KILL_COUNT env) in the specified application pod by sending SIGKILL termination signal to its docker socket (hence docker runtime is required)
    • Containers are killed using the kill command provided by pumba
    • Pumba is run as a pod on the application node. It have ability to kill the application containers multiple times. Which can be varied by TOTAL_CHAOS_DURATION and CHAOS_INTERVAL.
  • litmus chaoslib details
    • It can kill the multiple containers in the specified application pod by litmus Lib.
    • Containers are killed using the crictl stop command.
    • container-kill is run as a pod on the application node. It have ability to kill the application containers multiple times. Which can be varied by TOTAL_CHAOS_DURATION and CHAOS_INTERVAL.
  • Tests deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application
  • Good for testing recovery of pods having side-car containers

Integrations

  • Container kill is achieved using the pumba or litmus chaos library
  • The desired pumba and litmus image can be configured in the env variable LIB_IMAGE.

Steps to Execute the Chaos Experiment

  • This Chaos Experiment can be triggered by creating a ChaosEngine resource on the cluster. To understand the values to provide in a ChaosEngine specification, refer Getting Started

  • Follow the steps in the sections below to create the chaosServiceAccount, prepare the ChaosEngine & execute the experiment.

Prepare chaosServiceAccount

  • Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment.

Sample Rbac Manifest

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: container-kill-sa
  namespace: default
  labels:
    name: container-kill-sa
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: container-kill-sa
  namespace: default
  labels:
    name: container-kill-sa
rules:
- apiGroups: ["","litmuschaos.io","batch","apps"]
  resources: ["pods","jobs","pods/exec","pods/log","events","chaosengines","chaosexperiments","chaosresults"]
  verbs: ["create","list","get","patch","update","delete"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: container-kill-sa
  namespace: default
  labels:
    name: container-kill-sa
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: container-kill-sa
subjects:
- kind: ServiceAccount
  name: container-kill-sa
  namespace: default

Prepare ChaosEngine

  • Provide the application info in spec.appinfo
  • Override the experiment tunables if desired in experiments.spec.components.env
  • To understand the values to provided in a ChaosEngine specification, refer ChaosEngine Concepts

Supported Experiment Tunables

Variables Description Specify In ChaosEngine Notes
TARGET_CONTAINER The container to be killed inside the pod Mandatory If the TARGET_CONTAINER is not provided it will delete the first container
CHAOS_INTERVAL Time interval b/w two successive container kill (in sec) Optional If the CHAOS_INTERVAL is not provided it will take the default value of 10s
TOTAL_CHAOS_DURATION The time duration for chaos injection (seconds) Optional Defaults to 20s
LIB_IMAGE LIB Image used to kill the container Optional Defaults to `litmuschaos/go-runner:latest`
LIB The category of lib use to inject chaos Optional It can be pumba or litmus, pumba: docker runtime and litmus: containerd/crio runtime
RAMP_TIME Period to wait before injection of chaos in sec Optional
CONTAINER_PATH Path of the containerd socket file Optional Defaults to `/run/containerd/containerd.sock`
INSTANCE_ID A user-defined string that holds metadata/info about current run/instance of chaos. Ex: 04-05-2020-9-00. This string is appended as suffix in the chaosresult CR name. Optional Ensure that the overall length of the chaosresult CR is still < 64 characters

Sample ChaosEngine Manifest

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: nginx-chaos
  namespace: default
spec:
  # It can be true/false
  annotationCheck: 'true'
  # It can be active/stop
  engineState: 'active'
  #ex. values: ns1:name=percona,ns2:run=nginx 
  auxiliaryAppInfo: ''
  appinfo:
    appns: 'default'
    applabel: 'app=nginx'
    appkind: 'deployment'
  chaosServiceAccount: container-kill-sa
  monitoring: false
  # It can be delete/retain
  jobCleanUpPolicy: 'delete' 
  experiments:
    - name: container-kill
      spec:
        components:
          env:
            # specify the name of the container to be killed
            - name: TARGET_CONTAINER
              value: 'nginx'

            # provide the chaos interval
            - name: CHAOS_INTERVAL
              value: '10'

            # provide the total chaos duration
            - name: TOTAL_CHAOS_DURATION
              value: '20'

Create the ChaosEngine Resource

  • Create the ChaosEngine manifest prepared in the previous step to trigger the Chaos.

    kubectl apply -f chaosengine.yml

  • If the chaos experiment is not executed, refer to the troubleshooting section to identify the root cause and fix the issues.

Watch Chaos progress

  • View pod restart count by setting up a watch on the pods in the application namespace

    watch -n 1 kubectl get pods -n <application-namespace>

Abort/Restart the Chaos Experiment

  • To stop the pod-delete experiment immediately, either delete the ChaosEngine resource or execute the following command:

    kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"stop"}}'

  • To restart the experiment, either re-apply the ChaosEngine YAML or execute the following command:

    kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"active"}}'

Check Chaos Experiment Result

  • Check whether the application is resilient to the container kill, once the experiment (job) is completed. The ChaosResult resource name is derived like this: <ChaosEngine-Name>-<ChaosExperiment-Name>.

    kubectl describe chaosresult nginx-chaos-container-kill -n <application-namespace>

Application Container Kill Demo

  • A sample recording of this experiment execution is provided here.
← Pod DeletePod Network Latency →