PID insufficiency drill
Simulate PID exhaustion on an ECS instance to test service high availability when new processes cannot be created.
How it works
The PID insufficiency drill uses the ACS-ECS-TaskLimit Cloud Assistant plug-in to create processes with the fork() method until the smaller of the pid_max and threads-max values is reached, preventing new processes or threads from being created.
Procedure
Prerequisites
-
Cloud Assistant Agent is installed on the target ECS instance.
-
Cloud Assistant status is Normal on the instance. See Check Cloud Assistant status and troubleshoot exceptions.
Inject a fault
-
Log on to the ECS instance.
-
Run the
ACS-ECS-TaskLimitCloud Assistant plug-in with sudo privileges.ImportantThe
ACS-ECS-TaskLimitplug-in affects only new processes, not existing ones.sudo acs-plugin-manager --exec --plugin ACS-ECS-TaskLimit --params injectThe following output indicates that the
ACS-ECS-TaskLimitCloud Assistant plug-in has started.[ecs-user@iZ2zeioa ~]$ sudo acs-plugin-manager --exec --plugin ACS-ECS-TaskLimit --params inject Done Will create 28201 tasks to limit [ecs-user@iZ2zeioazolicrtw3ow42bZ ~]$ created 22548 threads in child -
Verify that the fault is injected.
If the fault is injected, new processes fail to start with the error:
-bash: fork: retry: Resource temporarily unavailable.[ecs-user@iZ2zeioazoli ~]$ sudo systemctl restart sysstat.service -bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: Resource temporarily unavailable
Recover from the fault
Remove the injected fault by using one of the following methods:
-
Recommended: Restart the instance.
-
If you can run commands as expected, run the following recovery command.
ImportantThe recovery command may fail due to system blockage.
sudo acs-plugin-manager --exec --plugin ACS-ECS-TaskLimit --params recover