next

Edit

CoreDNS Pod Delete Experiment Details

Experiment Metadata

Type Description Tested K8s Platform
CoreDNS CoreDNS pod delete experiment Kubeadm, Minikube

Prerequisites

  • Ensure that the Litmus ChaosOperator is running by executing kubectl get pods in operator namespace (typically, litmus). If not, install from here
  • Ensure that the coredns-pod-delete experiment resource is available in the cluster by executing kubectl get chaosexperiments in the desired namespace. If not, install from here

Entry Criteria

  • CoreDNS replicas are healthy before chaos injection
  • Service resolution works successfully as determined by deploying a sample nginx application and a custom liveness app querying the nginx application

Exit Criteria

  • CoreDNS replicas are healthy after chaos injection
  • Service resolution works successfully as determined by deploying a sample nginx application and a custom liveness app querying the nginx application

Details

  • Causes graceful pod failure of an coreDNS replicas
  • Tests deployment sanity (replica availability & uninterrupted service) and recovery workflow of the service
  • Service resolution will failed if coreDNS replicas are not present.

Integrations

  • Pod failures can be effected using one of these chaos libraries: litmus

Steps to Execute the ChaosExperiment

  • This ChaosExperiment can be triggered by creating a ChaosEngine resource on the cluster. To understand the values to provide in a ChaosEngine specification, refer Getting Started

  • Follow the steps in the sections below to create the chaosServiceAccount, prepare the ChaosEngine & execute the experiment.

Prepare chaosServiceAccount

  • Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment.

Sample Rbac Manifest

apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns-pod-delete-sa
  namespace: kube-system
  labels:
    name: coredns-pod-delete-sa
    app.kubernetes.io/part-of: litmus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: coredns-pod-delete-sa
  labels:
    name: coredns-pod-delete-sa
    app.kubernetes.io/part-of: litmus
rules:
- apiGroups: ["","litmuschaos.io","batch"]
  resources: ["services", "pods","jobs","events","pods/log","chaosengines","chaosexperiments","chaosresults"]
  verbs: ["create","list","get","patch","update","delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: coredns-pod-delete-sa
  labels:
    name: coredns-pod-delete-sa
    app.kubernetes.io/part-of: litmus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: coredns-pod-delete-sa
subjects:
- kind: ServiceAccount
  name: coredns-pod-delete-sa
  namespace: kube-system

Prepare ChaosEngine

  • Provide the application info in spec.appinfo

    • It will be default as 
        appinfo:
    appns: kube-system
    applabel: 'k8s-app=kube-dns'
    appkind: deployment

  • Override the experiment tunables if desired in experiments.spec.components.env

  • To understand the values to provided in a ChaosEngine specification, refer ChaosEngine Concepts

Supported Experiment Tunables

Variables Description Specify In ChaosEngine Notes
CHAOS_NAMESPACE This is chaos namespace which will create all infra chaos resources in that namespace Mandatory Default to `kube-system`
TOTAL_CHAOS_DURATION The time duration for chaos insertion (sec) Optional Defaults to 15s
CHAOS_INTERVAL Time interval b/w two successive pod failures (sec) Optional Defaults to 5s
KILL_COUNT No. of application pods to be deleted Optional Default to `1`, kill_count > 1 is only supported by litmus lib , not by the powerfulseal
LIB The chaos lib used to inject the chaos Optional Defaults to `litmus`, Supported: `litmus`
INSTANCE_ID A user-defined string that holds metadata/info about current run/instance of chaos. Ex: 04-05-2020-9-00. This string is appended as suffix in the chaosresult CR name. Optional Ensure that the overall length of the chaosresult CR is still < 64 characters

Sample ChaosEngine Manifest

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-coredns
  namespace: kube-system
spec:
  appinfo:
    appns: 'kube-system'
    applabel: 'k8s-app=kube-dns'
    appkind: 'deployment'
  # It can be active/stop
  engineState: 'active'
  #ex. values: ns1:name=percona,ns2:run=nginx 
  auxiliaryAppInfo: ''
  chaosServiceAccount: coredns-pod-delete-sa
  experiments:
    - name: coredns-pod-delete
      spec:
        components:
          env: 
            # set chaos duration (in sec) as desired
            - name: TOTAL_CHAOS_DURATION
              value: '30'
        
            # set chaos interval (in sec) as desired
            - name: CHAOS_INTERVAL
              value: '10'

            - name: CHAOS_NAMESPACE
              value: 'kube-system'

Create the ChaosEngine Resource

  • Create the ChaosEngine manifest prepared in the previous step to trigger the Chaos.

    kubectl apply -f chaosengine.yml

  • If the chaos experiment is not executed, refer to the troubleshooting section to identify the root cause and fix the issues.

Watch Chaos progress

  • View coredns pod terminations & recovery by setting up a watch on the coredns pods in the application namespace

    watch kubectl get pods -n kube-system

Check ChaosExperiment Result

  • Check whether the application is resilient to the coredns pod failure, once the experiment (job) is completed. The ChaosResult resource name is derived like this: <ChaosEngine-Name>-<ChaosExperiment-Name>.

    kubectl describe chaosresult engine-coredns-coredns-pod-delete -n <chaos-namespace>

← Broker Disk FailureCassandra Pod Delete →