Perform a disk I/O hang drill
Inject a disk I/O hang fault on an ECS instance and verify that your services can recover correctly.
Limitations
ECS instances must run Linux distributions compatible with cgroup v1, such as Alibaba Cloud Linux 3 and 2.
Implementation
The Cloud Assistant plug-in ACS-ECS-IoHang uses cgroups to throttle disk I/O. Cgroups are a Linux kernel mechanism that limits resource usage (CPU, memory, I/O, and network bandwidth) for one or more processes.
Procedure
Prerequisites
-
Cloud Assistant Agent is installed on the target ECS instance.
-
Cloud Assistant status is Normal on the instance. See Check Cloud Assistant status and troubleshoot exceptions.
Fault injection
-
Log on to the ECS instance.
-
Run the
ACS-ECS-IoHangplug-in as sudo:sudo acs-plugin-manager --exec --plugin ACS-ECS-IoHang --params inject,disk=vda,[duration=paramA]Parameters in brackets (
[]) are optional:-
disk (required): the target disk. Run
lsblkto view attached disks. Set toallto inject faults into all disks. -
duration (optional): fault injection duration. Unit: seconds. Default value: 300.
-
-
Verify that the fault injection succeeded.
-
The following output indicates successful injection:

-
Verify that business read/write speeds meet expectations.
-
Fault recovery
-
(Recommended) Wait for automatic recovery after timeout.
NoteFault injection times out after 300 seconds. For system disks, automatic recovery may not occur. In this case, forcefully restart the instance in the ECS console. See Restart an instance.
-
If faults are injected only to data disks, run the following command to recover:
sudo acs-plugin-manager --exec --plugin ACS-ECS-IoHang --params recover
Example
Run lsblk to view disks attached to the instance. This example uses the vdb data disk for fault injection.
-
To observe the drill effect, simulate reading data from the vdb disk. Skip this step if you use an actual business scenario.
sudo dd if=/dev/vdb of=/dev/nullQuery I/O usage with the
iotopcommand.NoteIf iotop is not installed, install it based on your Linux distribution:
-
Alibaba Cloud Linux 3 or 2 or CentOS 7
sudo yum install -y iotop -
Ubuntu or Debian
sudo apt install -y iotop
Total DISK READ : 110.05 M/s | Total DISK WRITE : 73.53 K/s Actual DISK READ: 110.05 M/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 2905 be/4 root 110.05 M/s 0.00 B/s 0.00 % 65.38 % dd if=/dev/vdb of=/dev/null 244 be/3 root 0.00 B/s 73.53 K/s 0.00 % 0.00 % systemd-journald 1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init noibrs splash 2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd] 4 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H] 6 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [mm_percpu_wq] 7 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0] 8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched] 9 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh] 10 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0] -
-
Inject the fault.
sudo acs-plugin-manager --exec --plugin ACS-ECS-IoHang --params inject,disk=vdb,duration=120The output shows the injection parameters and the major/minor device numbers of the specified disk, indicating success.
[ecs-user@iZ0jl5pl6qr ~]$ sudo acs-plugin-manager --exec --plugin ACS-ECS-IoHang --params inject,disk=vda params: disk:vda, duration:300 Processing disk with MAJ:MIN = 253:0 Done -
Verify the fault injection effect.
Run
iotopto check I/O usage. The disk read speed drops to 0 B/s.Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init noibrs splash 2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd] 4 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H] 5 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/u4:0] 6 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [mm_percpu_wq] 7 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0] 8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched] 9 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh] 10 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0] -
Wait for fault recovery.
After the fault injection times out, the disk read speed of the simulated business process recovers.
Total DISK READ : 109.94 M/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 109.70 M/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 1486 be/4 root 109.94 M/s 0.00 B/s 0.00 % 67.51 % dd if=/dev/vdb of=/dev/null 2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init noibrs splash 2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd] 4 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H] 5 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/u4:0] 6 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [mm_percpu_wq] 7 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0] 8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched] 9 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh] 10 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0]