Fault drills

Updated at:

Inject controllable faults into ECS instances with Cloud Assistant plugins (ecs-fault-{scenename} or ACS-ECS-{scenename}) to verify high availability, test recovery mechanisms, and reduce MTTR.

Benefits

  • Free and open source: Executed plugins are saved in the Cloud Assistant directory, including fault injection source code and executable files.

  • Scenario-based: Each plugin targets one drill scenario. Download and use plugins based on your needs.

  • Convenient and efficient: Any ECS instance with Cloud Assistant installed can run drill plugins. A single command installs and executes a plugin.

Supported drill scenarios

Drill scenario

Description

Downtime drill

Server downtime caused by software or hardware abnormalities, and is virtually unavoidable. Simulate kernel faults on an ECS instance to cause downtime, to test your business system's response to downtime, verify recovery capabilities and monitoring alerts, and can then develop response strategies to reduce the risk of business interruption in production.

High CPU utilization drill

Excessively high CPU utilization can cause latency or service interruption. Inject high CPU utilization faults into an ECS instance to test system response under specific CPU loads, verify recovery capabilities and monitoring alerts, and develop response strategies to reduce the risk of business interruption in production.

Out-of-memory drill

Inject a process that continuously consumes memory on an ECS instance to simulate an out-of-memory (OOM) condition. Test whether processes terminate as expected, verify recovery capabilities and monitoring alerts, and develop response strategies to reduce the risk of business interruption in production.

Network drills

  • Network packet loss drill: Network congestion, hardware failures, and link interference can cause packet loss. This drill verifies system alerts and recovery mechanisms when packet loss occurs.

  • Network interruption drill: Hardware link issues, carrier fluctuations, and configuration errors can cause prolonged network outages. This drill verifies monitoring and recovery capabilities when a node becomes unavailable.

  • Network delay drill: Increased traffic and unstable lines can cause high network latency, degrading application performance. This drill verifies system alerts and recovery mechanisms when network delay occurs.

Disk and I/O drills

  • High disk capacity utilization drill: Data accumulation and temporary file buildup can exhaust disk capacity, causing performance degradation, crashes, and data loss. This drill verifies system stability and data integrity when disk utilization is high or full.

  • Disk I/O hang drill: I/O hangs occur when read or write operations stall due to factors including hardware failures, driver issues, file system errors, or network congestion, causing performance degradation and data inconsistency. This drill verifies system alerts and recovery mechanisms when a disk I/O hang occurs.

  • High disk I/O load drill: High I/O load can result from excessive process loads, unexpected non-business process usage, or insufficient memory, causing performance degradation and data loss. This drill verifies system alerts and recovery mechanisms under high disk I/O load.

System resource drills

  • High system load drills: System load measures the average number of runnable and uninterruptible processes over a time interval. Monitoring load helps detect overloads early and trigger alerts.

  • PID insufficiency drills: A process identifier (PID) uniquely identifies a process and is reused after the process terminates. Although rare, PID exhaustion prevents new processes from being created and can suspend services. This drill simulates PID exhaustion to test service high availability.

  • System time jump drills: A time jump is a sudden change in the system clock that can disrupt time-sensitive services such as logs and synchronization backups. This drill tests whether the system can promptly restore correct time and recover services after a time jump.

Procedure

Fault drills use Cloud Assistant plugins. Commands follow this syntax:

  • Fault injection

    sudo acs-plugin-manager --exec --plugin {plugin-name} --params inject,paramA=a,paramB=b
  • Fault recovery

    sudo acs-plugin-manager --exec --plugin {plugin-name} --params recover
Note
  • Replace {plugin-name} with the actual plugin name.

  • Plugins support two actions: injection (inject) and recovery (recover).

  • Injection parameters are comma-separated key-value pairs in key=value format.