Simulate an out-of-memory event

Updated at:

Trigger an OOM event on an ECS instance to verify process termination, alerting, and recovery.

Key concepts

OOM and the OOM killer

When memory is exhausted, the Linux kernel activates the OOM killer. It scores each running process (oom_score) and terminates those with low priority and high memory usage first to free memory and prevent crashes.

How the drill works

The drill uses the Cloud Assistant plugin ACS-ECS-HighMemory:

  1. Calculates memory to allocate based on the target percentage.

  2. Starts the trigger_oom process, which consumes memory at a specified rate until reaching the target.

  3. If percent is omitted, triggers an OOM event directly. If percent is specified, holds memory at the target level for the configured duration.

Adjust oom_score_adj to control which process the OOM killer terminates:

Value

Behavior

-1000

Prevents the OOM killer from terminating the process

0 (default)

Neutral priority

1000

Process is terminated first

Set a negative score for the injection process so the OOM killer targets the business process instead.

Prerequisites

Ensure the following:

  • Cloud Assistant Agent is installed on the ECS instance.

  • The Cloud Assistant status is Normal on the instance.

See View the status of Cloud Assistant and handle anomalies.

Inject a fault

  1. Log on to the ECS instance. See Log on to a Linux instance using Workbench.

  2. Run ACS-ECS-HighMemory with sudo privileges. Example: Set memory usage to 90% for 120 seconds:

    The square brackets [] indicate optional parameters. Omit the brackets when running the command.

    Parameters

    Parameter

    Required

    Description

    Default

    score

    No

    The oom_score_adj for the injection process. Range: -1000 to 1000. A negative value (e.g., -100) makes the OOM killer more likely to terminate the business process. Decrease further if needed.

    0

    percent

    No

    Target memory usage as a percentage. If omitted, an OOM event is triggered directly.

    Triggers OOM

    rate

    No

    Memory allocation rate in MB/s.

    0 (no limit)

    duration

    No

    Duration in seconds to maintain target memory usage. Memory is released after this period.

    300

       sudo acs-plugin-manager --exec --plugin ACS-ECS-HighMemory --params inject,[score=paramA],[percent=paramB],[rate=paramC],[duration=paramD]
       sudo acs-plugin-manager --exec --plugin ACS-ECS-HighMemory --params inject,percent=90,duration=120
  3. Verify the injection. Without an OOM event Run top to check whether memory usage reaches the target, or check the monitoring page in the ECS console or CloudMonitor console. With an OOM event Search the system log: If the output contains Out of memory, the fault was injected successfully. Verify the terminated process matches your expectation. If not, adjust score.

       dmesg -T | grep "Out of memory"

    Memory usage in CloudMonitor

    OOM event in dmesg output

Recover from the fault

If you specified percent, use one of these methods:

  • Run the recovery command (recommended): Verify that memory drops to the pre-injection level after running:

      sudo acs-plugin-manager --exec --plugin ACS-ECS-HighMemory --params recover
  • Wait for automatic release: Memory is released after the duration timeout (default: 300 seconds).

If you did not specify percent, the system triggers an OOM event and usually recovers automatically. You may need to restart the instance to prevent other processes from being unexpectedly terminated.

Example: 90% memory drill

This example injects a fault with 90% memory usage, a 20 MB/s allocation rate, and a 120-second duration.

  1. Inject the fault:

       sudo acs-plugin-manager --exec --plugin ACS-ECS-HighMemory --params inject,percent=90,rate=20,duration=120

    Successful injection output

  2. Verify with top. Run top, press M to sort by memory, and m to show the memory bar. The output shows 90.2% memory usage, with trigger_oom consuming 84.4%.

    top command showing memory usage

  3. Observe automatic recovery. After 120 seconds, memory is released and usage returns to the pre-drill level.

    Memory usage after recovery

  4. Verify in the console. The monitoring chart shows memory increasing at the specified rate, reaching 90%, holding for 120 seconds, then dropping to the pre-injection level.

    Instance monitoring showing memory spike

Troubleshooting