Simulate an out-of-memory event
Trigger an OOM event on an ECS instance to verify process termination, alerting, and recovery.
Key concepts
OOM and the OOM killer
When memory is exhausted, the Linux kernel activates the OOM killer. It scores each running process (oom_score) and terminates those with low priority and high memory usage first to free memory and prevent crashes.
How the drill works
The drill uses the Cloud Assistant plugin ACS-ECS-HighMemory:
-
Calculates memory to allocate based on the target percentage.
-
Starts the
trigger_oomprocess, which consumes memory at a specified rate until reaching the target. -
If
percentis omitted, triggers an OOM event directly. Ifpercentis specified, holds memory at the target level for the configured duration.
Adjust oom_score_adj to control which process the OOM killer terminates:
|
Value |
Behavior |
|
-1000 |
Prevents the OOM killer from terminating the process |
|
0 (default) |
Neutral priority |
|
1000 |
Process is terminated first |
Set a negative score for the injection process so the OOM killer targets the business process instead.
Prerequisites
Ensure the following:
-
Cloud Assistant Agent is installed on the ECS instance.
-
The Cloud Assistant status is Normal on the instance.
See View the status of Cloud Assistant and handle anomalies.
Inject a fault
-
Log on to the ECS instance. See Log on to a Linux instance using Workbench.
-
Run
ACS-ECS-HighMemorywith sudo privileges. Example: Set memory usage to 90% for 120 seconds:The square brackets
[]indicate optional parameters. Omit the brackets when running the command.Parameters
Parameter
Required
Description
Default
scoreNo
The
oom_score_adjfor the injection process. Range: -1000 to 1000. A negative value (e.g., -100) makes the OOM killer more likely to terminate the business process. Decrease further if needed.0
percentNo
Target memory usage as a percentage. If omitted, an OOM event is triggered directly.
Triggers OOM
rateNo
Memory allocation rate in MB/s.
0 (no limit)
durationNo
Duration in seconds to maintain target memory usage. Memory is released after this period.
300
sudo acs-plugin-manager --exec --plugin ACS-ECS-HighMemory --params inject,[score=paramA],[percent=paramB],[rate=paramC],[duration=paramD]sudo acs-plugin-manager --exec --plugin ACS-ECS-HighMemory --params inject,percent=90,duration=120 -
Verify the injection. Without an OOM event Run
topto check whether memory usage reaches the target, or check the monitoring page in the ECS console or CloudMonitor console. With an OOM event Search the system log: If the output containsOut of memory, the fault was injected successfully. Verify the terminated process matches your expectation. If not, adjustscore.dmesg -T | grep "Out of memory"

Recover from the fault
If you specified percent, use one of these methods:
-
Run the recovery command (recommended): Verify that memory drops to the pre-injection level after running:
sudo acs-plugin-manager --exec --plugin ACS-ECS-HighMemory --params recover -
Wait for automatic release: Memory is released after the
durationtimeout (default: 300 seconds).
If you did not specify percent, the system triggers an OOM event and usually recovers automatically. You may need to restart the instance to prevent other processes from being unexpectedly terminated.
Example: 90% memory drill
This example injects a fault with 90% memory usage, a 20 MB/s allocation rate, and a 120-second duration.
-
Inject the fault:
sudo acs-plugin-manager --exec --plugin ACS-ECS-HighMemory --params inject,percent=90,rate=20,duration=120
-
Verify with
top. Runtop, pressMto sort by memory, andmto show the memory bar. The output shows 90.2% memory usage, withtrigger_oomconsuming 84.4%.
-
Observe automatic recovery. After 120 seconds, memory is released and usage returns to the pre-drill level.

-
Verify in the console. The monitoring chart shows memory increasing at the specified rate, reaching 90%, holding for 120 seconds, then dropping to the pre-injection level.

Troubleshooting
-
High memory usage can cause stuttering and slow responses. See What do I do if the memory usage of a Linux instance is high?.
-
OOM events may result from insufficient memory or resource exhaustion due to frequent requests. See How do I handle OOM issues in a Linux instance?.
