next

Edit

Azure Instance Stop Experiment Details

Experiment Metadata

Type Description Tested K8s Platform
Azure Power off of an azure instance for a certain chaos duration EKS, AKS, K3s

Prerequisites

  • Ensure that Kubernetes Version > 1.16
  • Ensure that the Litmus Chaos Operator is running by executing kubectl get pods in operator namespace (typically, litmus). If not, install from here
  • Ensure that the azure-instance-stop experiment resource is available in the cluster by executing kubectl get chaosexperiments in the desired namespace If not, install from here
  • Ensure that you have sufficient Azure access to stop and start the an instance.
  • We will use azure file-based authentication to connect with the instance using azure GO SDK in the experiment. For generating auth file run az ad sp create-for-rbac --sdk-auth > azure.auth Azure CLI command.
  • Ensure to create a Kubernetes secret having the auth file created in the step in CHAOS_NAMESPACE. A sample secret file looks like:
apiVersion: v1
kind: Secret
metadata:
  name: cloud-secret
type: Opaque
stringData:
  azure.auth: |-
    {
      "clientId": "XXXXXXXXX",
      "clientSecret": "XXXXXXXXX",
      "subscriptionId": "XXXXXXXXX",
      "tenantId": "XXXXXXXXX",
      "activeDirectoryEndpointUrl": "XXXXXXXXX",
      "resourceManagerEndpointUrl": "XXXXXXXXX",
      "activeDirectoryGraphResourceId": "XXXXXXXXX",
      "sqlManagementEndpointUrl": "XXXXXXXXX",
      "galleryEndpointUrl": "XXXXXXXXX",
      "managementEndpointUrl": "XXXXXXXXX"
    }
  • If you change the secret key name (from azure.auth) please also update the AZURE_AUTH_LOCATION ENV value on experiment.yamlwith the same name.

Entry-Criteria

  • Azure instance is healthy before chaos injection.

Exit-Criteria

  • Azure instance is healthy post chaos injection.

Details

  • Causes PowerOff an Azure instance before bringing it back to running state after the specified chaos duration.
  • It helps to check the performance of the application/process running on the instance.

Integrations

  • Azure Instance Stop can be effected using the chaos library: litmus, which makes use of azure sdk to start/stop an instance.
  • The desired chaoslib can be selected by setting the above options as value for the env variable LIB

Steps to Execute the Chaos Experiment

  • This Chaos Experiment can be triggered by creating a ChaosEngine resource on the cluster. To understand the values to provide in a ChaosEngine specification, refer Getting Started

  • Follow the steps in the sections below to create the chaosServiceAccount, prepare the ChaosEngine & execute the experiment.

Prepare chaosServiceAccount

  • Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment.

Sample Rbac Manifest

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: azure-instance-stop-sa
  namespace: default
  labels:
    name: azure-instance-stop-sa
    app.kubernetes.io/part-of: litmus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: azure-instance-stop-sa
  labels:
    name: azure-instance-stop-sa
    app.kubernetes.io/part-of: litmus
rules:
- apiGroups: [""]
  resources: ["pods","events","secrets"]
  verbs: ["create","list","get","patch","update","delete","deletecollection"]
- apiGroups: [""]
  resources: ["pods/exec","pods/log"]
  verbs: ["create","list","get"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["create","list","get","delete","deletecollection"]
- apiGroups: ["litmuschaos.io"]
  resources: ["chaosengines","chaosexperiments","chaosresults"]
  verbs: ["create","list","get","patch","update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: azure-instance-stop-sa
  labels:
    name: azure-instance-stop-sa
    app.kubernetes.io/part-of: litmus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: azure-instance-stop-sa
subjects:
- kind: ServiceAccount
  name: azure-instance-stop-sa
  namespace: default

Prepare ChaosEngine

  • Provide the application info in spec.appinfo. It is an optional parameter for infra level experiment.
  • Provide the auxiliary applications info (ns & labels) in spec.auxiliaryAppInfo
  • Override the experiment tunables if desired in experiments.spec.components.env
  • To understand the values to provided in a ChaosEngine specification, refer ChaosEngine Concepts

Supported Experiment Tunables

Variables Description Specify In ChaosEngine Notes
AZURE_INSTANCE_NAME Instance name(s) of the target azure instance (comma separated if multiple). Mandatory For AKS ndoes, the instance name is from the scale set section in Azure and not the node name from AKS node pool
RESOURCE_GROUP The resource group of the target instance Mandatory
SCALE_SET Whether instance is part of Scale set Mandatory Accepts "enable"/"disable". Default is "disable"
TOTAL_CHAOS_DURATION The time duration for chaos insertion (sec) Optional Defaults to 30s
CHAOS_INTERVAL The interval (in sec) between successive instance poweroff. Optional Defaults to 30s
RAMP_TIME Period to wait before injection of chaos in sec Optional
SEQUENCE It defines sequence of chaos execution for multiple target pods Optional Default value: parallel. Supported: serial, parallel
INSTANCE_ID A user-defined string that holds metadata/info about current run/instance of chaos. Ex: 04-05-2020-9-00. This string is appended as suffix in the chaosresult CR name. Optional Ensure that the overall length of the chaosresult CR is still < 64 characters

Sample ChaosEngine Manifest

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: nginx-chaos
  namespace: default
spec:
  engineState: 'active'
  chaosServiceAccount: azure-instance-stop-sa
  experiments:
    - name: azure-instance-stop
      spec:
        components:
          env: 
            # set chaos duration (in sec) as desired
            - name: TOTAL_CHAOS_DURATION
              value: '30'

            # set chaos intreval (in sec) as desired
            - name: CHAOS_INTERVAL
              value: '30'            

            # provide the target instance name(s) (comma separated if multiple)
            - name: AZURE_INSTANCE_NAME
              value: ''

            # provide the resource group of the instance
            - name: RESOURCE_GROUP
              value: ''
            
            # accepts enable/disable, default is disable
            - name: SCALE_SET
              value: ''

Create the ChaosEngine Resource

  • Create the ChaosEngine manifest prepared in the previous step to trigger the Chaos.

    kubectl apply -f chaosengine.yml

  • If the chaos experiment is not executed, refer to the troubleshooting section to identify the root cause and fix the issues.

Watch Chaos progress

  • Monitor the azure state from Azure CLI.

    az vm list -d -o table --query "[?name=='<vm-name>']" (vm name without <>)

  • You can also use Azure console to keep a watch over the instance state.

Abort/Restart the Chaos Experiment

  • To stop the azure-instance-terminate experiment immediately, either delete the ChaosEngine resource or execute the following command:

    kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"stop"}}'

  • To restart the experiment, either re-apply the ChaosEngine YAML or execute the following command:

    kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"active"}}'

Check Chaos Experiment Result

  • Check whether the application is resilient to the azure-instance-stop, once the experiment (job) is completed. The ChaosResult resource name is derived like this: <ChaosEngine-Name>-<ChaosExperiment-Name>.

    kubectl describe chaosresult nginx-chaos-azure-instance-stop -n <chaos-namespace>

Azure Instance Stop Experiment Demo

  • A sample recording of this experiment execution will be added soon.
← VM PowerOffTarget Container Failure →