Network packet loss drill
Packet loss is a common network issue that can be caused by network congestion, hardware failures, or link interference. You can use this drill to test how your system sends alerts and recovers when packet loss occurs.
Limitations
This drill requires the tc (Traffic Control) tool and its dependent kernel component, netem.
-
If your system lacks the tc tool, run
sudo yum install -y iproute-tcorsudo apt-get install -y iproute2. Alternatively, you can specify a parameter to install it automatically when you run the plugin. -
If your system kernel lacks the netem component, which is common on some CentOS versions, run
sudo yum install kernel-modules-extrato install the package and then restart the instance.WarningInstalling
kernel-modules-extramodifies the kernel version. Proceed with caution. We recommend that you use an ECS instance that runs a different operating system for this drill.
How it works
This solution uses the Cloud Assistant plugin ACS-ECS-NetLoss. The plugin uses the tc (Traffic Control) utility and the netem component to add specific rules to a network interface, which enables traffic control within the Linux kernel. You can configure the plugin to affect all IP addresses or a single target IP address. This drill does not affect the Cloud Assistant network segment 100.100.0.0/16.
Procedure
Prerequisites
-
Cloud Assistant Agent is installed on the target ECS instance.
-
Cloud Assistant status is Normal on the instance. See Check Cloud Assistant status and troubleshoot exceptions.
Inject the fault
-
Log on to the ECS instance.
For more information, see Log on to a Linux instance using Workbench.
-
As a user with sudo privileges, run the Cloud Assistant plugin
ACS-ECS-NetLoss.sudo acs-plugin-manager --exec --plugin ACS-ECS-NetLoss --params inject,dev=eth0,[rate=paramA],[target-ip=paramB],[replace=paramC],[duration=paramD],[install-tc=paramF]The following list describes the fault injection parameters. Optional parameters are enclosed in brackets (
[]).-
dev (Required): The network interface for fault injection, such as
eth0. You can runifconfigto view the available network interfaces on your system. -
rate (Optional): The packet loss percentage. Default value: 10%.
-
target-ip (Optional): The target IP address to affect. By default, this parameter is empty, and the packet loss rule applies to all IP addresses. If you specify a target IP address, the drill affects only traffic to or from that IP address.
-
replace (Optional): Set this parameter to
trueto overwrite existing tc rules on the network interface. Otherwise, the injection fails if conflicting rules exist. -
duration (Optional): The duration of the fault injection in seconds. Default value: 300.
-
install-tc (Optional): If the
tcutility is missing, set this parameter totrueto automatically install it. Default value:false.The following output shows that the
ACS-ECS-NetLossCloud Assistant plugin ran successfully.[ecs-user@iZ0j1fp0t**** ~]$ sudo acs-plugin-manager --exec --plugin ACS-ECS-NetLoss --params inject,dev=eth0,rate=40 Will add a 40% network packet loss to dev: eth0 Done
-
-
Verify that the fault injection was successful.
Ping a target host to check for packet loss. In the following example, the output shows an average packet loss of approximately 44.4%.
[ecs-user@iZ0jlfp0bytxxx ~]$ ping www.taobao.com PING www.taobao.com.danuoyi.tbcache.com (180.xxx.xxx.xxx) 56(84) bytes of data. 64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=2 ttl=55 time=12.5 ms 64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=3 ttl=55 time=11.6 ms 64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=5 ttl=55 time=11.6 ms 64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=6 ttl=55 time=12.4 ms 64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=7 ttl=55 time=12.5 ms 64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=10 ttl=55 time=18.1 ms 64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=11 ttl=55 time=12.1 ms 64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=12 ttl=55 time=11.9 ms 64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=14 ttl=55 time=11.6 ms 64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=21 ttl=55 time=11.6 ms 64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=22 ttl=55 time=11.7 ms 64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=23 ttl=55 time=11.8 ms 64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=24 ttl=55 time=12.5 ms 64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=25 ttl=55 time=12.5 ms 64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=27 ttl=55 time=12.6 ms ^C --- www.taobao.com.danuoyi.tbcache.com ping statistics --- 27 packets transmitted, 15 received, 44.4444% packet loss, time 26333ms rtt min/avg/max/mdev = 11.615/12.502/18.083/1.551 ms
Recover from the fault
-
Method 1: Wait for the drill to end. The fault is automatically removed when the duration expires.
-
Method 2: Manually run the recovery command.
sudo acs-plugin-manager --exec --plugin ACS-ECS-NetLoss --params recover