High I/O load fault drill

Updated at:

A high I/O load is a common issue that can be caused by factors like high process loads, unexpected background processes, or insufficient memory. It may degrade service performance and lead to risks such as data loss. You can use this fault drill to simulate a high I/O load and test your system's alerting and recovery capabilities.

How it works

This fault drill uses the Cloud Assistant plug-in ACS-ECS-HighIo, which internally uses the dd (data duplicator) tool to copy and convert files to increase the disk I/O load:

  • In read mode, the plug-in first writes a temporary file to the destination path and then repeatedly reads the file to generate a high read load.

  • In write mode, the plug-in continuously writes to a file to generate a high write load.

Procedure

Prerequisites

Inject the fault

  1. Log on to the ECS instance.

    For more information, see Log on to a Linux instance using Workbench.

  2. As a user with sudo permissions, run the ACS-ECS-HighIo Cloud Assistant plug-in.

    sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params inject,[model=paramA],[dir=paramB],[bs=paramC],[count=paramD],[duration=paramE]

    The parameters in brackets ([]) are optional:

    • model (Optional): The I/O mode for the fault drill. Valid values: read (default) and write.

    • dir (Optional): The destination path. You can specify a path to stress-test a specific disk. Default: the system's root directory.

    • bs (Optional): The block size for each read/write operation, formatted like the bs parameter of the dd command. The default is 1M to help saturate I/O.

    • count (Optional): The number of blocks in a single read loop. Default: 1024.

    • duration (Optional): The duration of the fault drill, in seconds. Default: 300.

    The following output indicates that the ACS-ECS-HighIo Cloud Assistant plug-in ran successfully.

    [ecs-user@iZ0jl5pl6qr9          ~]$ sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params inject
    params: model-read, dir=/, bs=1M, count=1024, duration=300
    Done
    /usr/local/share/aliyun-assist/plugin/ACS-ECS-HighIo/TMP_FILE_PATH.tmp
  3. Run the iotop command to verify the fault injection.

    The following output indicates a successful fault injection.

    Total DISK READ :      105.85 M/s | Total DISK WRITE :        0.00 B/s
    Actual DISK READ:      105.85 M/s | Actual DISK WRITE:        0.00 B/s
      TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
     3270 be/4 root       105.85 M/s    0.00 B/s  0.00 % 99.16 % dd if=//AliFaultHighio.tmp of=/dev/null bs=1M count=1024 iflag=dsync,direct,fullblock
        1 be/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % init noibrs splash
        2 be/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
        4 be/0 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H]
        6 be/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [mm_percpu_wq]
        7 be/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
        8 be/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_sched]
        9 be/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_bh]
       10 rt/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
       11 rt/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
       12 be/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [cpuhp/0]
       13 be/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [cpuhp/1]
       14 rt/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/1]
       15 rt/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
       16 be/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]
       17 be/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/1:0]
       18 be/0 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/1:0H]
       19 be/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [kdevtmpfs]
       20 be/0 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [netns]
       21 be/4 root         0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_tasks_kthre]

Recover from the fault

Run the recovery command on the ECS instance and verify that the I/O level returns to its pre-drill state.

Important

You must run this command after the fault drill completes to delete the temporary file.

sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params recover

Example

  1. Inject a fault that performs a read operation on the /usr path for 60 seconds.

    sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params inject,dir=/usr,duration=60

    The output displays the fault injection parameters, the result, and the path of the temporary file.

    [ecs-user@iZ0jl5pl6qr9ecxxx ~]$ sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params inject,dir=/usr,duration=60
    params: model=read, dir=/usr, bs=1M, count=1024, duration=60
    Done
    /usr/local/share/aliyun-assist/plugin/ACS-ECS-HighIo/TMP_FILE_PATH.tmp
  2. In the /usr directory, check for the AliFaultHighio.tmp temporary file.

    [ecs-user@iZ2zeioazo         ~]$ cd /usr
    [ecs-user@iZ2zeioazo         usr]$ ll
    total 512128
    -rw-r--r--.  1 root root 524288000 Jul 31 10:38 AliFaultHighio.tmp
    dr-xr-xr-x.  2 root root     32768 May 28 11:17 bin
    drwxr-xr-x.  2 root root      4096 Feb  9  2022 games
    drwxr-xr-x. 35 root root      4096 May 28 11:16 include
    dr-xr-xr-x. 35 root root      4096 May 28 11:08 lib
    dr-xr-xr-x. 49 root root     36864 May 28 11:16 lib64
    drwxr-xr-x. 30 root root      4096 May 28 11:13 libexec
    drwxr-xr-x. 14 root root      4096 May 28 11:13 local
    dr-xr-xr-x.  2 root root     20480 Jul 31 10:27 sbin
    dr-xr-xr-x. 94 root root      4096 May 28 11:16 share
    drwxr-xr-x.  4 root root      4096 May 28 11:00 src
    lrwxrwxrwx.  1 root root        10 Feb  9  2022 tmp -> ./var/tmp
    [ecs-user@iZ2zeioazolicrtw3ow42bZ usr]$
  3. Run the iotop command to check the disk I/O status.

    Total DISK READ :   106.05 M/s | Total DISK WRITE :       0.00 B/s
    Actual DISK READ:   106.05 M/s | Actual DISK WRITE:      31.72 K/s
      TID  PRIO  USER     DISK READ  DISK WRITE>  COMMAND
      365 be/3 root        0.00 B/s    0.00 B/s [jbd2/vda3-8]
     3443 be/4 root      106.05 M/s    0.00 B/s dd if=/usr/AliFaultHighio.tmp of=/dev/null bs=1M count=1024 iflag=dsync,direct,fullblock
  4. Run the fault recovery command.

    sudo acs-plugin-manager --exec --plugin ACS-ECS-HighIo --params recover