High I/O load fault drill
A high I/O load is a common issue that can be caused by factors like high process loads, unexpected background processes, or insufficient memory. It may degrade service performance and lead to risks such as data loss. You can use this fault drill to simulate a high I/O load and test your system's alerting and recovery capabilities.
How it works
This fault drill uses the Cloud Assistant plug-in ACS-ECS-HighIo, which internally uses the dd (data duplicator) tool to copy and convert files to increase the disk I/O load:
-
In read mode, the plug-in first writes a temporary file to the destination path and then repeatedly reads the file to generate a high read load.
-
In write mode, the plug-in continuously writes to a file to generate a high write load.
Procedure
Prerequisites
-
Cloud Assistant Agent is installed on the target ECS instance.
-
Cloud Assistant status is Normal on the instance. See Check Cloud Assistant status and troubleshoot exceptions.
Inject the fault
-
Log on to the ECS instance.
For more information, see Log on to a Linux instance using Workbench.
-
As a user with
sudopermissions, run theACS-ECS-HighIoCloud Assistant plug-in.sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params inject,[model=paramA],[dir=paramB],[bs=paramC],[count=paramD],[duration=paramE]The parameters in brackets (
[]) are optional:-
model(Optional): The I/O mode for the fault drill. Valid values:read(default) andwrite. -
dir(Optional): The destination path. You can specify a path to stress-test a specific disk. Default: the system's root directory. -
bs(Optional): The block size for each read/write operation, formatted like thebsparameter of theddcommand. The default is1Mto help saturate I/O. -
count(Optional): The number of blocks in a single read loop. Default:1024. -
duration(Optional): The duration of the fault drill, in seconds. Default:300.
The following output indicates that the
ACS-ECS-HighIoCloud Assistant plug-in ran successfully.[ecs-user@iZ0jl5pl6qr9 ~]$ sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params inject params: model-read, dir=/, bs=1M, count=1024, duration=300 Done /usr/local/share/aliyun-assist/plugin/ACS-ECS-HighIo/TMP_FILE_PATH.tmp -
-
Run the
iotopcommand to verify the fault injection.The following output indicates a successful fault injection.
Total DISK READ : 105.85 M/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 105.85 M/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 3270 be/4 root 105.85 M/s 0.00 B/s 0.00 % 99.16 % dd if=//AliFaultHighio.tmp of=/dev/null bs=1M count=1024 iflag=dsync,direct,fullblock 1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init noibrs splash 2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd] 4 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H] 6 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [mm_percpu_wq] 7 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0] 8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched] 9 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh] 10 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0] 11 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/0] 12 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [cpuhp/0] 13 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [cpuhp/1] 14 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/1] 15 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/1] 16 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/1] 17 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/1:0] 18 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/1:0H] 19 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kdevtmpfs] 20 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [netns] 21 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_tasks_kthre]
Recover from the fault
Run the recovery command on the ECS instance and verify that the I/O level returns to its pre-drill state.
You must run this command after the fault drill completes to delete the temporary file.
sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params recover
Example
-
Inject a fault that performs a read operation on the
/usrpath for 60 seconds.sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params inject,dir=/usr,duration=60The output displays the fault injection parameters, the result, and the path of the temporary file.
[ecs-user@iZ0jl5pl6qr9ecxxx ~]$ sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params inject,dir=/usr,duration=60 params: model=read, dir=/usr, bs=1M, count=1024, duration=60 Done /usr/local/share/aliyun-assist/plugin/ACS-ECS-HighIo/TMP_FILE_PATH.tmp -
In the
/usrdirectory, check for theAliFaultHighio.tmptemporary file.[ecs-user@iZ2zeioazo ~]$ cd /usr [ecs-user@iZ2zeioazo usr]$ ll total 512128 -rw-r--r--. 1 root root 524288000 Jul 31 10:38 AliFaultHighio.tmp dr-xr-xr-x. 2 root root 32768 May 28 11:17 bin drwxr-xr-x. 2 root root 4096 Feb 9 2022 games drwxr-xr-x. 35 root root 4096 May 28 11:16 include dr-xr-xr-x. 35 root root 4096 May 28 11:08 lib dr-xr-xr-x. 49 root root 36864 May 28 11:16 lib64 drwxr-xr-x. 30 root root 4096 May 28 11:13 libexec drwxr-xr-x. 14 root root 4096 May 28 11:13 local dr-xr-xr-x. 2 root root 20480 Jul 31 10:27 sbin dr-xr-xr-x. 94 root root 4096 May 28 11:16 share drwxr-xr-x. 4 root root 4096 May 28 11:00 src lrwxrwxrwx. 1 root root 10 Feb 9 2022 tmp -> ./var/tmp [ecs-user@iZ2zeioazolicrtw3ow42bZ usr]$ -
Run the
iotopcommand to check the disk I/O status.Total DISK READ : 106.05 M/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 106.05 M/s | Actual DISK WRITE: 31.72 K/s TID PRIO USER DISK READ DISK WRITE> COMMAND 365 be/3 root 0.00 B/s 0.00 B/s [jbd2/vda3-8] 3443 be/4 root 106.05 M/s 0.00 B/s dd if=/usr/AliFaultHighio.tmp of=/dev/null bs=1M count=1024 iflag=dsync,direct,fullblock -
Run the fault recovery command.
sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params recover