Scale the application replicas and test the node autoscaling on cluster
Experiment Metadata
Type | Description | Tested K8s Platform |
---|---|---|
Generic | Scale the application replicas and test the node autoscaling on cluster | GKE, EKS, Minikube |
Prerequisites
- Ensure that the Litmus Chaos Operator is running by executing
kubectl get pods
in operator namespace (typically,litmus
). If not, install from here - Ensure that the
pod-autoscaler
experiment resource is available in the cluster by executingkubectl get chaosexperiments
in the desired namespace. If not, install from here
Entry Criteria
- Application pods are healthy on the respective Nodes before chaos injection
Exit Criteria
- Application pods will be healthy post chaos injection
Details
The experiment aims to check the ability of nodes to accommodate the number of replicas a given application pod.
This experiment can be used for other scenarios as well, such as for checking the Node auto-scaling feature. For example, check if the pods are successfully rescheduled within a specified period in cases where the existing nodes are already running at the specified limits.
Integrations
- Pod Autoscaler can be effected using the chaos library:
litmus
- The desired chaos library can be selected by setting
litmus
as value for the env variableLIB
Steps to Execute the Chaos Experiment
This Chaos Experiment can be triggered by creating a ChaosEngine resource on the cluster. To understand the values to provide in a ChaosEngine specification, refer Getting Started
Follow the steps in the sections below to create the chaosServiceAccount, prepare the ChaosEngine & execute the experiment.
Prepare chaosServiceAccount
- Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment.
Sample Rbac Manifest
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: pod-autoscaler-sa
namespace: default
labels:
name: pod-autoscaler-sa
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: pod-autoscaler-sa
labels:
name: pod-autoscaler-sa
rules:
- apiGroups: ["","litmuschaos.io","batch","apps"]
resources: ["pods","deployments","jobs","events","chaosengines","pods/log","chaosexperiments","chaosresults"]
verbs: ["create","list","get","patch","update","delete"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get","list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: pod-autoscaler-sa
labels:
name: pod-autoscaler-sa
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: pod-autoscaler-sa
subjects:
- kind: ServiceAccount
name: pod-autoscaler-sa
namespace: default
Prepare ChaosEngine
- Provide the application info in
spec.appinfo
- Provide the auxiliary applications info (ns & labels) in
spec.auxiliaryAppInfo
- Override the experiment tunables if desired in
experiments.spec.components.env
- To understand the values to provided in a ChaosEngine specification, refer ChaosEngine Concepts
Supported Experiment Tunables
Variables | Description | Specify In ChaosEngine | Notes |
---|---|---|---|
TOTAL_CHAOS_DURATION | The timeout for the chaos experiment (in seconds) | Optional | Defaults to 60 |
LIB | The chaos lib used to inject the chaos | Optional | Defaults to `litmus` |
REPLICA_COUNT | Number of replicas upto which we want to scale | Mandatory | nil |
RAMP_TIME | Period to wait before and after injection of chaos in sec | Optional | |
INSTANCE_ID | A user-defined string that holds metadata/info about current run/instance of chaos. Ex: 04-05-2020-9-00. This string is appended as suffix in the chaosresult CR name. | Optional | Ensure that the overall length of the chaosresult CR is still < 64 characters |
Sample ChaosEngine Manifest
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: nginx-chaos
namespace: default
spec:
# It can be true/false
annotationCheck: 'false'
# It can be active/stop
engineState: 'active'
#ex. values: ns1:name=percona,ns2:run=nginx
auxiliaryAppInfo: ''
appinfo:
appns: 'default'
applabel: 'app=nginx'
appkind: 'deployment'
chaosServiceAccount: pod-autoscaler-sa
monitoring: false
# It can be delete/retain
jobCleanUpPolicy: 'delete'
experiments:
- name: pod-autoscaler
spec:
components:
env:
# set chaos duration (in sec) as desired
- name: TOTAL_CHAOS_DURATION
value: '60'
# number of replicas to scale
- name: REPLICA_COUNT
value: '5'
Create the ChaosEngine Resource
Create the ChaosEngine manifest prepared in the previous step to trigger the Chaos.
kubectl apply -f chaosengine.yml
If the chaos experiment is not executed, refer to the troubleshooting section to identify the root cause and fix the issues.
Watch Chaos progress
Setting up a watch of the number of repicas scaling and its state in the Kubernetes Cluster
watch kubectl get pods -n <namespace>
Check Chaos Experiment Result
Check whether the nodes can accomodate the application pods in the given timeout(Total Chaos Duration) , once the experiment (job) is completed. The ChaosResult resource name is derived like this:
<ChaosEngine-Name>-<ChaosExperiment-Name>
.kubectl describe chaosresult nginx-chaos-pod-autoscaler -n <application-namespace>
Pod Autoscaler Experiment Demo
- A sample recording of this experiment execution will be provided soon.