Network delay drill
Network delay directly affects the response time of applications and services. High network delay, caused by increased network traffic or unstable connections, can degrade the user experience. Use this drill to test your system's alerting and recovery capabilities when network delay occurs.
Limitations
This drill requires the tc (Traffic Control) tool and its dependent kernel component, netem.
-
If your system lacks the tc tool, run
sudo yum install -y iproute-tcorsudo apt-get install -y iproute2. Alternatively, you can specify a parameter to install it automatically when you run the plugin. -
If your system kernel lacks the netem component, which is common on some CentOS versions, run
sudo yum install kernel-modules-extrato install the package and then restart the instance.WarningInstalling
kernel-modules-extramodifies the kernel version. Proceed with caution. We recommend that you use an ECS instance that runs a different operating system for this drill.
How it works
This solution uses the Cloud Assistant plugin ACS-ECS-NetDelay. The plugin uses the tc (Traffic Control) tool and the netem component to add rules to a NIC, enabling traffic control within the Linux kernel. The Cloud Assistant CIDR block 100.100.0.0/16 is not affected.
Procedure
Prerequisites
-
Cloud Assistant Agent is installed on the target ECS instance.
-
Cloud Assistant status is Normal on the instance. See Check Cloud Assistant status and troubleshoot exceptions.
Fault injection
-
Log on to the ECS instance.
For more information, see Log on to a Linux instance using Workbench.
-
As a user with sudo privileges, run the Cloud Assistant plugin
ACS-ECS-NetDelay.sudo acs-plugin-manager --exec --plugin ACS-ECS-NetDelay --params inject,dev=eth0,[time=paramA],[jitter=paramB],[target-ip=paramC],[replace=paramD],[duration=paramE],[install-tc=paramF]Parameters in brackets (
[]) are optional. The following list describes each parameter:-
dev (Required): The target NIC for fault injection, such as eth0. You can run
ifconfigto view the NICs on your system. -
time (Optional): The delay time in milliseconds (ms). Default: 100.
-
jitter (Optional): The jitter value in ms. Default: 10.
-
target-ip (Optional): The target IP address to affect. If unspecified, the delay applies to all IP addresses. Otherwise, only traffic to the specified IP is affected.
-
replace (Optional): The injection may fail if the NIC already has conflicting tc rules. To avoid this, set this parameter to true to overwrite the existing rules.
-
duration (Optional): The duration of the fault in seconds (s). Default: 300.
-
install-tc (Optional): If the
tctool is missing from your system, set this parameter totrueto install it automatically. Default:false.
The following output indicates a successful run of the
ACS-ECS-NetDelayCloud Assistant plugin.Will add a 100ms delay and a 10ms jitter to dev: eth0 Done -
-
Ping the target network to verify the fault injection.
Fault recovery
-
Method 1: Wait for the fault to recover automatically when the specified duration expires.
-
Method 2: Manually run the fault recovery command.
sudo acs-plugin-manager --exec --plugin ACS-ECS-NetDelay --params recover
Drill example
-
Simulate business traffic to a target network.
ping www.taobao.comThis example uses www.taobao.com for demonstration purposes. The following is a sample output:
[ecs-user@iZ0j... ~]$ ping www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (xxx) 56(84) bytes of data. 64 bytes from xxx: icmp_seq=1 ttl=53 time=11.9 ms 64 bytes from xxx: icmp_seq=2 ttl=53 time=11.10 ms 64 bytes from xxx: icmp_seq=3 ttl=53 time=11.8 ms 64 bytes from xxx: icmp_seq=4 ttl=53 time=11.9 ms 64 bytes from xxx: icmp_seq=5 ttl=53 time=12.0 ms 64 bytes from xxx: icmp_seq=6 ttl=53 time=11.9 ms 64 bytes from xxx: icmp_seq=7 ttl=53 time=12.4 ms 64 bytes from xxx: icmp_seq=8 ttl=53 time=13.1 ms 64 bytes from xxx: icmp_seq=9 ttl=53 time=11.8 ms 64 bytes from xxx: icmp_seq=10 ttl=53 time=11.9 ms -
Inject the fault.
sudo acs-plugin-manager --exec --plugin ACS-ECS-NetDelay --params inject,dev=eth0,time=1000,jitter=30,duration=120This command adds a 1,000 ms delay and a 30 ms jitter to all traffic on the NIC.
-
Check the effect of the injection.
After injecting the fault, the ping output shows that the delay increases by approximately 1,000 ms, fluctuating within the specified 1,000 ± 30 ms range.
64 bytes from xxx: icmp_seq=126 ttl=53 time=11.10 ms 64 bytes from xxx: icmp_seq=127 ttl=53 time=12.1 ms 64 bytes from xxx: icmp_seq=128 ttl=53 time=11.9 ms 64 bytes from xxx: icmp_seq=129 ttl=53 time=1070 ms 64 bytes from xxx: icmp_seq=130 ttl=53 time=969 ms 64 bytes from xxx: icmp_seq=131 ttl=53 time=1033 ms 64 bytes from xxx: icmp_seq=132 ttl=53 time=1039 ms 64 bytes from xxx: icmp_seq=133 ttl=53 time=1028 ms 64 bytes from xxx: icmp_seq=134 ttl=53 time=995 ms 64 bytes from xxx: icmp_seq=135 ttl=53 time=975 ms 64 bytes from xxx: icmp_seq=136 ttl=53 time=1032 ms 64 bytes from xxx: icmp_seq=137 ttl=53 time=992 ms 64 bytes from xxx: icmp_seq=138 ttl=53 time=1015 ms 64 bytes from xxx: icmp_seq=139 ttl=53 time=1027 ms 64 bytes from xxx: icmp_seq=140 ttl=53 time=1024 ms 64 bytes from xxx: icmp_seq=141 ttl=53 time=1046 ms 64 bytes from xxx: icmp_seq=142 ttl=53 time=982 ms 64 bytes from xxx: icmp_seq=143 ttl=53 time=1068 ms 64 bytes from xxx: icmp_seq=144 ttl=53 time=984 ms 64 bytes from xxx: icmp_seq=145 ttl=53 time=989 ms -
Wait for fault recovery.
When the specified duration expires, the delay returns to normal and the network recovers.
64 bytes from xxx : icmp_seq=268 ttl=55 time=977 ms 64 bytes from xxx : icmp_seq=269 ttl=55 time=1066 ms 64 bytes from xxx : icmp_seq=270 ttl=55 time=946 ms 64 bytes from xxx : icmp_seq=271 ttl=55 time=1023 ms 64 bytes from xxx : icmp_seq=272 ttl=55 time=928 ms 64 bytes from xxx : icmp_seq=273 ttl=55 time=945 ms 64 bytes from xxx : icmp_seq=274 ttl=55 time=1049 ms 64 bytes from xxx : icmp_seq=275 ttl=55 time=1014 ms 64 bytes from xxx : icmp_seq=276 ttl=55 time=11.8 ms 64 bytes from xxx : icmp_seq=277 ttl=55 time=11.8 ms 64 bytes from xxx : icmp_seq=278 ttl=55 time=11.7 ms 64 bytes from xxx : icmp_seq=279 ttl=55 time=11.7 ms 64 bytes from xxx : icmp_seq=280 ttl=55 time=11.6 ms 64 bytes from xxx : icmp_seq=281 ttl=55 time=11.6 ms 64 bytes from xxx : icmp_seq=282 ttl=55 time=11.6 ms 64 bytes from xxx : icmp_seq=283 ttl=55 time=11.7 ms