Chaos Workflows with Argo and LitmusChaos
When simulating real-world failures via chaos injection on development/staging environments as part of a left-shifted, continuous validation strategy, it is preferable to construct potential failure sequence over executing standalone chaos injection actions. Often, this translates into failures during a certain workload condition (such as, say, percentage load), multiple (parallel) failures of dependent & independent services etc.
Chaos Workflows are a set of actions strung together to achieve desired chaos impact on a Kubernetes cluster. They are an effective mechanism to simulate real world conditions & gauge application behaviour in an effective manner.
This document specifies the procedure to setup and execute a simple chaos workflow to execute a pod-kill chaos on an nginx deployment while a benchmark run is in progress.
Install Argo Workflow Infrastructure
The Argo workflow infrastructure consists of the Argo workflow CRDs, Workflow Controller, associated RBAC & Argo CLI. The steps are shown below to install Argo in the standard cluster-wide mode, where the workflow controller operates on all namespaces. Ensure that you have the right permission to be able to create the said resources.
Create argo namespace
kubectl create ns argo
Create the CRDs, workflow controller deployment with associated RBAC
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo/stable/manifests/install.yaml -n argo
Install the argo CLI on the test harness machine (where the kubeconfig is available)
curl -sLO https://github.com/argoproj/argo/releases/download/v2.8.0/argo-linux-amd64
chmod +x argo-linux-amd64
mv ./argo-linux-amd64 /usr/local/bin/argo
Install a Sample Application: Nginx
Install a simple multi-replica stateless Nginx deployment with service exposed over nodeport
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/chaos-workflows/master/App/nginx.yaml
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/chaos-workflows/master/App/service.yaml
Install Litmus Infrastructure
Apply the LitmusChaos Operator manifest:
kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v1.7.0.yaml
Install the litmus-admin service account to be used by the chaos-operator while executing the experiment (this example uses the admin-mode of chaos execution)
Install the RBAC & experiment CR for litmus
kubectl apply -f https://litmuschaos.github.io/litmus/litmus-admin-rbac.yaml
- Install the pod-delete chaos experiment
kubectl apply -f https://hub.litmuschaos.io/api/chaos/master?file=charts/generic/pod-delete/experiment.yaml -n litmus
Note: If you are interested in using chaostoolkit to perform the pod-delete, instead of the native litmus lib, you can apply this rbac & experiment manifests instead of the ones described above.
Create the service account and associated RBAC, which will be used by the Argo workflow controller to execute the actions specified in the workflow. In our case, this corresponds to the launch of the Nginx benchmark job and creating the chaosengine to trigger the pod-delete chaos action. In our example, we place it in the namespace where the litmus chaos resources reside, i.e., litmus.
kubectl apply -f https://raw.githubusercontent.com/litmuschaos/chaos-workflows/master/Argo/argo-access.yaml -n litmus
Create the Chaos Workflow
Applying the workflow manifest performs the following actions in parallel:
- Starts an Nginx benchmark job for the specified duration (300s)
- Triggers a random pod-kill of the Nginx replica by creating the chaosengine CR.
- Cleans up after chaos.
argo submit https://raw.githubusercontent.com/litmuschaos/chaos-workflows/master/Argo/argowf-native-pod-delete.yaml -n litmus
- Note: If you are using the chaostoolkit experiment, submit this workflow manifest instead.
Visualize the Chaos Workflow
You can visualize the progress of the chaos workflow via the Argo UI. Convert the argo-server service to type NodePort & view the dashboard at https://<node-ip>:<nodeport>
kubectl patch svc argo-server -n argo -p '{"spec": {"type": "NodePort"}}'