Perform a disk I/O hang drill

Updated at:

Inject a disk I/O hang fault on an ECS instance and verify that your services can recover correctly.

Limitations

ECS instances must run Linux distributions compatible with cgroup v1, such as Alibaba Cloud Linux 3 and 2.

Implementation

The Cloud Assistant plug-in ACS-ECS-IoHang uses cgroups to throttle disk I/O. Cgroups are a Linux kernel mechanism that limits resource usage (CPU, memory, I/O, and network bandwidth) for one or more processes.

Procedure

Prerequisites

Fault injection

  1. Log on to the ECS instance.

    See Log on to a Linux instance using Workbench.

  2. Run the ACS-ECS-IoHang plug-in as sudo:

    sudo acs-plugin-manager --exec --plugin ACS-ECS-IoHang --params inject,disk=vda,[duration=paramA]

    Parameters in brackets ([]) are optional:

    • disk (required): the target disk. Run lsblk to view attached disks. Set to all to inject faults into all disks.

    • duration (optional): fault injection duration. Unit: seconds. Default value: 300.

  3. Verify that the fault injection succeeded.

    • The following output indicates successful injection:

      image

    • Verify that business read/write speeds meet expectations.

Fault recovery

  • (Recommended) Wait for automatic recovery after timeout.

    Note

    Fault injection times out after 300 seconds. For system disks, automatic recovery may not occur. In this case, forcefully restart the instance in the ECS console. See Restart an instance.

  • If faults are injected only to data disks, run the following command to recover:

    sudo acs-plugin-manager --exec --plugin ACS-ECS-IoHang --params recover

Example

Run lsblk to view disks attached to the instance. This example uses the vdb data disk for fault injection.

  1. To observe the drill effect, simulate reading data from the vdb disk. Skip this step if you use an actual business scenario.

    sudo dd if=/dev/vdb of=/dev/null

    Query I/O usage with the iotop command.

    Note

    If iotop is not installed, install it based on your Linux distribution:

    1. Alibaba Cloud Linux 3 or 2 or CentOS 7

      sudo yum install -y iotop
    2. Ubuntu or Debian

      sudo apt install -y iotop
    Total DISK READ :     110.05 M/s | Total DISK WRITE :       73.53 K/s
    Actual DISK READ:     110.05 M/s | Actual DISK WRITE:        0.00 B/s
      TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
     2905 be/4 root      110.05 M/s    0.00 B/s  0.00 % 65.38 % dd if=/dev/vdb of=/dev/null
      244 be/3 root        0.00 B/s   73.53 K/s  0.00 %  0.00 % systemd-journald
        1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init noibrs splash
        2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
        4 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H]
        6 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [mm_percpu_wq]
        7 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
        8 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_sched]
        9 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_bh]
       10 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
  2. Inject the fault.

    sudo acs-plugin-manager --exec --plugin ACS-ECS-IoHang --params inject,disk=vdb,duration=120

    The output shows the injection parameters and the major/minor device numbers of the specified disk, indicating success.

    [ecs-user@iZ0jl5pl6qr    ~]$ sudo acs-plugin-manager --exec --plugin ACS-ECS-IoHang --params inject,disk=vda
    params: disk:vda, duration:300
    Processing disk with MAJ:MIN = 253:0
    Done
  3. Verify the fault injection effect.

    Run iotop to check I/O usage. The disk read speed drops to 0 B/s.

    Total DISK READ :       0.00 B/s | Total DISK WRITE :       0.00 B/s
    Actual DISK READ:       0.00 B/s | Actual DISK WRITE:       0.00 B/s
      TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
        1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init noibrs splash
        2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
        4 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H]
        5 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/u4:0]
        6 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [mm_percpu_wq]
        7 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
        8 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_sched]
        9 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_bh]
       10 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
  4. Wait for fault recovery.

    After the fault injection times out, the disk read speed of the simulated business process recovers.

    Total DISK READ :     109.94 M/s | Total DISK WRITE :       0.00 B/s
    Actual DISK READ:     109.70 M/s | Actual DISK WRITE:       0.00 B/s
      TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
     1486 be/4 root      109.94 M/s    0.00 B/s  0.00 % 67.51 % dd if=/dev/vdb of=/dev/null
        2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init noibrs splash
        2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
        4 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H]
        5 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/u4:0]
        6 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [mm_percpu_wq]
        7 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
        8 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_sched]
        9 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_bh]
       10 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]