Simulate a downtime drill

Updated at:

Simulate kernel faults on an ECS instance to test recovery, verify monitoring alerts, and develop response strategies.

How it works

The ecs-fault-oscrash Cloud Assistant plugin triggers a kernel panic through the sysrq module. The system then automatically restarts and resumes normal operation.

Important

This drill may affect your business. Ensure your system has high availability (HA) capabilities and kdump is enabled. See How do I enable the kdump service on a Linux instance?

Procedure

Prerequisites

Inject a fault

  1. Connect to the ECS instance as a user with sudo privileges. See Log on to a Linux instance using Workbench.

  2. Run the ecs-fault-oscrash Cloud Assistant plugin:

    sudo acs-plugin-manager --exec --plugin ecs-fault-oscrash --params inject

    The following output indicates that the ecs-fault-oscrash plugin ran successfully.

    image

  3. Verify the fault injection.

    • If an Instance Restart Due To Instance Error event appears in unexpected O&M events, the fault was injected.

      image

    • Run the uptime command on the instance to check the system runtime and confirm whether a restart occurred.

      The following output shows the instance restarted at 18:21:46, confirming fault injection.

      image

Recover from the fault

The ECS instance automatically restarts and resumes normal operation after the drill. If the instance fails to restart, force restart it in the ECS console. See Restart an instance.