PID insufficiency drill

Updated at:

Simulate PID exhaustion on an ECS instance to test service high availability when new processes cannot be created.

How it works

The PID insufficiency drill uses the ACS-ECS-TaskLimit Cloud Assistant plug-in to create processes with the fork() method until the smaller of the pid_max and threads-max values is reached, preventing new processes or threads from being created.

Procedure

Prerequisites

Inject a fault

  1. Log on to the ECS instance.

    See Log on to a Linux instance using Workbench.

  2. Run the ACS-ECS-TaskLimit Cloud Assistant plug-in with sudo privileges.

    Important

    The ACS-ECS-TaskLimit plug-in affects only new processes, not existing ones.

    sudo acs-plugin-manager --exec --plugin ACS-ECS-TaskLimit --params inject

    The following output indicates that the ACS-ECS-TaskLimit Cloud Assistant plug-in has started.

    [ecs-user@iZ2zeioa ~]$ sudo acs-plugin-manager --exec --plugin ACS-ECS-TaskLimit --params inject
    Done
    Will create 28201 tasks to limit
    [ecs-user@iZ2zeioazolicrtw3ow42bZ ~]$ created 22548 threads in child
  3. Verify that the fault is injected.

    If the fault is injected, new processes fail to start with the error: -bash: fork: retry: Resource temporarily unavailable.

    [ecs-user@iZ2zeioazoli        ~]$ sudo systemctl restart sysstat.service
    -bash: fork: retry: Resource temporarily unavailable
    -bash: fork: retry: Resource temporarily unavailable
    -bash: fork: retry: Resource temporarily unavailable
    -bash: fork: retry: Resource temporarily unavailable
    -bash: fork: retry: Resource temporarily unavailable
    -bash: fork: Resource temporarily unavailable

Recover from the fault

Remove the injected fault by using one of the following methods:

  • Recommended: Restart the instance.

  • If you can run commands as expected, run the following recovery command.

    Important

    The recovery command may fail due to system blockage.

    sudo acs-plugin-manager --exec --plugin ACS-ECS-TaskLimit --params recover