Fault drills
Inject controllable faults into ECS instances with Cloud Assistant plugins (ecs-fault-{scenename} or ACS-ECS-{scenename}) to verify high availability, test recovery mechanisms, and reduce MTTR.
Benefits
-
Free and open source: Executed plugins are saved in the Cloud Assistant directory, including fault injection source code and executable files.
-
Scenario-based: Each plugin targets one drill scenario. Download and use plugins based on your needs.
-
Convenient and efficient: Any ECS instance with Cloud Assistant installed can run drill plugins. A single command installs and executes a plugin.
Supported drill scenarios
|
Drill scenario |
Description |
|
Server downtime caused by software or hardware abnormalities, and is virtually unavoidable. Simulate kernel faults on an ECS instance to cause downtime, to test your business system's response to downtime, verify recovery capabilities and monitoring alerts, and can then develop response strategies to reduce the risk of business interruption in production. |
|
|
Excessively high CPU utilization can cause latency or service interruption. Inject high CPU utilization faults into an ECS instance to test system response under specific CPU loads, verify recovery capabilities and monitoring alerts, and develop response strategies to reduce the risk of business interruption in production. |
|
|
Inject a process that continuously consumes memory on an ECS instance to simulate an out-of-memory (OOM) condition. Test whether processes terminate as expected, verify recovery capabilities and monitoring alerts, and develop response strategies to reduce the risk of business interruption in production. |
|
|
Network drills |
|
|
Disk and I/O drills |
|
|
System resource drills |
|
Procedure
Fault drills use Cloud Assistant plugins. Commands follow this syntax:
-
Fault injection
sudo acs-plugin-manager --exec --plugin {plugin-name} --params inject,paramA=a,paramB=b -
Fault recovery
sudo acs-plugin-manager --exec --plugin {plugin-name} --params recover
-
Replace
{plugin-name}with the actual plugin name. -
Plugins support two actions: injection (inject) and recovery (recover).
-
Injection parameters are comma-separated key-value pairs in
key=valueformat.