Simulate high system load

Updated at:

Inject a high system load fault on an ECS instance with Cloud Assistant to test service resilience under stress.

How it works

The ACS-ECS-HighLoad Cloud Assistant plug-in runs the vfork command to create processes that raise the system load to a target value. Each process sleeps until timeout and exits, with minimal impact on system operations.

Procedure

Prerequisites

Inject a fault

  1. Log on to the ECS instance. See Log on to a Linux instance using Workbench.

  2. Run the ACS-ECS-HighLoad plug-in with sudo privileges:

    sudo acs-plugin-manager --exec --plugin ACS-ECS-HighLoad --params inject,[num-processes=paramA],[duration=paramB]

    Optional parameters enclosed in brackets []: o

    • num-processes: The number of processes to create. Roughly approximates the target system load. Default value: 100.

    • duration: The duration in seconds. Default value: 300.

    The following output shows that the ACS-ECS-HighLoad plug-in runs successfully.

    image

  3. Verify the fault injection.

    • Run the top command to check the 1-minute, 5-minute, and 15-minute load averages.

    • View the load curve in the Cloud Monitor console.

R ecover from the fault

  • Method 1 (recommended): Wait for automatic recovery after timeout.

  • Method 2: Run the recovery command on the ECS instance:

    sudo acs-plugin-manager --exec --plugin ACS-ECS-HighLoad --params recover

Example

  1. Inject a fault:

    sudo acs-plugin-manager --exec --plugin ACS-ECS-HighLoad --params inject

    The following output confirms fault injection.

    image

  2. Verify the fault injection effect.

    • Run the top command to view load averages. The following figure shows the 1-minute average is 98.33, the 5-minute average is 58.24, and the 15-minute average is 32.66.image

    • View load averages in the Cloud Monitor console.

      image

  3. Recover from the fault:

    • Wait for the load to return to normal after timeout. The following figure shows the 1-minute average dropping below the 5-minute average, indicating recovery.

      image

    • Run the recovery command:

      sudo acs-plugin-manager --exec --plugin ACS-ECS-HighLoad --params recover