Network packet loss drill

Updated at:

Packet loss is a common network issue that can be caused by network congestion, hardware failures, or link interference. You can use this drill to test how your system sends alerts and recovers when packet loss occurs.

Limitations

This drill requires the tc (Traffic Control) tool and its dependent kernel component, netem.

  • If your system lacks the tc tool, run sudo yum install -y iproute-tc or sudo apt-get install -y iproute2. Alternatively, you can specify a parameter to install it automatically when you run the plugin.

  • If your system kernel lacks the netem component, which is common on some CentOS versions, run sudo yum install kernel-modules-extra to install the package and then restart the instance.

    Warning

    Installing kernel-modules-extra modifies the kernel version. Proceed with caution. We recommend that you use an ECS instance that runs a different operating system for this drill.

How it works

This solution uses the Cloud Assistant plugin ACS-ECS-NetLoss. The plugin uses the tc (Traffic Control) utility and the netem component to add specific rules to a network interface, which enables traffic control within the Linux kernel. You can configure the plugin to affect all IP addresses or a single target IP address. This drill does not affect the Cloud Assistant network segment 100.100.0.0/16.

Procedure

Prerequisites

Inject the fault

  1. Log on to the ECS instance.

    For more information, see Log on to a Linux instance using Workbench.

  2. As a user with sudo privileges, run the Cloud Assistant plugin ACS-ECS-NetLoss.

    sudo acs-plugin-manager --exec --plugin ACS-ECS-NetLoss --params inject,dev=eth0,[rate=paramA],[target-ip=paramB],[replace=paramC],[duration=paramD],[install-tc=paramF]

    The following list describes the fault injection parameters. Optional parameters are enclosed in brackets ([]).

    • dev (Required): The network interface for fault injection, such as eth0. You can run ifconfig to view the available network interfaces on your system.

    • rate (Optional): The packet loss percentage. Default value: 10%.

    • target-ip (Optional): The target IP address to affect. By default, this parameter is empty, and the packet loss rule applies to all IP addresses. If you specify a target IP address, the drill affects only traffic to or from that IP address.

    • replace (Optional): Set this parameter to true to overwrite existing tc rules on the network interface. Otherwise, the injection fails if conflicting rules exist.

    • duration (Optional): The duration of the fault injection in seconds. Default value: 300.

    • install-tc (Optional): If the tc utility is missing, set this parameter to true to automatically install it. Default value: false.

      The following output shows that the ACS-ECS-NetLoss Cloud Assistant plugin ran successfully.

      [ecs-user@iZ0j1fp0t**** ~]$ sudo acs-plugin-manager --exec --plugin ACS-ECS-NetLoss --params inject,dev=eth0,rate=40
      Will add a 40% network packet loss to dev: eth0
      Done
  3. Verify that the fault injection was successful.

    Ping a target host to check for packet loss. In the following example, the output shows an average packet loss of approximately 44.4%.

    [ecs-user@iZ0jlfp0bytxxx            ~]$ ping www.taobao.com
    PING www.taobao.com.danuoyi.tbcache.com (180.xxx.xxx.xxx) 56(84) bytes of data.
    64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=2 ttl=55 time=12.5 ms
    64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=3 ttl=55 time=11.6 ms
    64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=5 ttl=55 time=11.6 ms
    64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=6 ttl=55 time=12.4 ms
    64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=7 ttl=55 time=12.5 ms
    64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=10 ttl=55 time=18.1 ms
    64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=11 ttl=55 time=12.1 ms
    64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=12 ttl=55 time=11.9 ms
    64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=14 ttl=55 time=11.6 ms
    64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=21 ttl=55 time=11.6 ms
    64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=22 ttl=55 time=11.7 ms
    64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=23 ttl=55 time=11.8 ms
    64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=24 ttl=55 time=12.5 ms
    64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=25 ttl=55 time=12.5 ms
    64 bytes from 180.xxx.xxx.xxx (180.xxx.xxx.xxx): icmp_seq=27 ttl=55 time=12.6 ms
    ^C
    --- www.taobao.com.danuoyi.tbcache.com ping statistics ---
    27 packets transmitted, 15 received, 44.4444% packet loss, time 26333ms
    rtt min/avg/max/mdev = 11.615/12.502/18.083/1.551 ms

Recover from the fault

  • Method 1: Wait for the drill to end. The fault is automatically removed when the duration expires.

  • Method 2: Manually run the recovery command.

    sudo acs-plugin-manager --exec --plugin ACS-ECS-NetLoss --params recover