Network delay drill

Updated at:

Network delay directly affects the response time of applications and services. High network delay, caused by increased network traffic or unstable connections, can degrade the user experience. Use this drill to test your system's alerting and recovery capabilities when network delay occurs.

Limitations

This drill requires the tc (Traffic Control) tool and its dependent kernel component, netem.

  • If your system lacks the tc tool, run sudo yum install -y iproute-tc or sudo apt-get install -y iproute2. Alternatively, you can specify a parameter to install it automatically when you run the plugin.

  • If your system kernel lacks the netem component, which is common on some CentOS versions, run sudo yum install kernel-modules-extra to install the package and then restart the instance.

    Warning

    Installing kernel-modules-extra modifies the kernel version. Proceed with caution. We recommend that you use an ECS instance that runs a different operating system for this drill.

How it works

This solution uses the Cloud Assistant plugin ACS-ECS-NetDelay. The plugin uses the tc (Traffic Control) tool and the netem component to add rules to a NIC, enabling traffic control within the Linux kernel. The Cloud Assistant CIDR block 100.100.0.0/16 is not affected.

Procedure

Prerequisites

Fault injection

  1. Log on to the ECS instance.

    For more information, see Log on to a Linux instance using Workbench.

  2. As a user with sudo privileges, run the Cloud Assistant plugin ACS-ECS-NetDelay.

    sudo acs-plugin-manager --exec --plugin ACS-ECS-NetDelay --params inject,dev=eth0,[time=paramA],[jitter=paramB],[target-ip=paramC],[replace=paramD],[duration=paramE],[install-tc=paramF]

    Parameters in brackets ([]) are optional. The following list describes each parameter:

    • dev (Required): The target NIC for fault injection, such as eth0. You can run ifconfig to view the NICs on your system.

    • time (Optional): The delay time in milliseconds (ms). Default: 100.

    • jitter (Optional): The jitter value in ms. Default: 10.

    • target-ip (Optional): The target IP address to affect. If unspecified, the delay applies to all IP addresses. Otherwise, only traffic to the specified IP is affected.

    • replace (Optional): The injection may fail if the NIC already has conflicting tc rules. To avoid this, set this parameter to true to overwrite the existing rules.

    • duration (Optional): The duration of the fault in seconds (s). Default: 300.

    • install-tc (Optional): If the tc tool is missing from your system, set this parameter to true to install it automatically. Default: false.

    The following output indicates a successful run of the ACS-ECS-NetDelay Cloud Assistant plugin.

    Will add a 100ms delay and a 10ms jitter to dev: eth0
    Done
  3. Ping the target network to verify the fault injection.

Fault recovery

  • Method 1: Wait for the fault to recover automatically when the specified duration expires.

  • Method 2: Manually run the fault recovery command.

    sudo acs-plugin-manager --exec --plugin ACS-ECS-NetDelay --params recover

Drill example

  1. Simulate business traffic to a target network.

    ping www.taobao.com

    This example uses www.taobao.com for demonstration purposes. The following is a sample output:

    [ecs-user@iZ0j...         ~]$ ping www.taobao.com
    PING www.taobao.com.danuoyi.tbcache.com (xxx) 56(84) bytes of data.
    64 bytes from xxx: icmp_seq=1 ttl=53 time=11.9 ms
    64 bytes from xxx: icmp_seq=2 ttl=53 time=11.10 ms
    64 bytes from xxx: icmp_seq=3 ttl=53 time=11.8 ms
    64 bytes from xxx: icmp_seq=4 ttl=53 time=11.9 ms
    64 bytes from xxx: icmp_seq=5 ttl=53 time=12.0 ms
    64 bytes from xxx: icmp_seq=6 ttl=53 time=11.9 ms
    64 bytes from xxx: icmp_seq=7 ttl=53 time=12.4 ms
    64 bytes from xxx: icmp_seq=8 ttl=53 time=13.1 ms
    64 bytes from xxx: icmp_seq=9 ttl=53 time=11.8 ms
    64 bytes from xxx: icmp_seq=10 ttl=53 time=11.9 ms
  2. Inject the fault.

    sudo acs-plugin-manager --exec --plugin ACS-ECS-NetDelay --params inject,dev=eth0,time=1000,jitter=30,duration=120

    This command adds a 1,000 ms delay and a 30 ms jitter to all traffic on the NIC.

  3. Check the effect of the injection.

    After injecting the fault, the ping output shows that the delay increases by approximately 1,000 ms, fluctuating within the specified 1,000 ± 30 ms range.

    64 bytes from xxx: icmp_seq=126 ttl=53 time=11.10 ms
    64 bytes from xxx: icmp_seq=127 ttl=53 time=12.1 ms
    64 bytes from xxx: icmp_seq=128 ttl=53 time=11.9 ms
    64 bytes from xxx: icmp_seq=129 ttl=53 time=1070 ms
    64 bytes from xxx: icmp_seq=130 ttl=53 time=969 ms
    64 bytes from xxx: icmp_seq=131 ttl=53 time=1033 ms
    64 bytes from xxx: icmp_seq=132 ttl=53 time=1039 ms
    64 bytes from xxx: icmp_seq=133 ttl=53 time=1028 ms
    64 bytes from xxx: icmp_seq=134 ttl=53 time=995 ms
    64 bytes from xxx: icmp_seq=135 ttl=53 time=975 ms
    64 bytes from xxx: icmp_seq=136 ttl=53 time=1032 ms
    64 bytes from xxx: icmp_seq=137 ttl=53 time=992 ms
    64 bytes from xxx: icmp_seq=138 ttl=53 time=1015 ms
    64 bytes from xxx: icmp_seq=139 ttl=53 time=1027 ms
    64 bytes from xxx: icmp_seq=140 ttl=53 time=1024 ms
    64 bytes from xxx: icmp_seq=141 ttl=53 time=1046 ms
    64 bytes from xxx: icmp_seq=142 ttl=53 time=982 ms
    64 bytes from xxx: icmp_seq=143 ttl=53 time=1068 ms
    64 bytes from xxx: icmp_seq=144 ttl=53 time=984 ms
    64 bytes from xxx: icmp_seq=145 ttl=53 time=989 ms
  4. Wait for fault recovery.

    When the specified duration expires, the delay returns to normal and the network recovers.

    64 bytes from xxx                          : icmp_seq=268 ttl=55 time=977 ms
    64 bytes from xxx                          : icmp_seq=269 ttl=55 time=1066 ms
    64 bytes from xxx                          : icmp_seq=270 ttl=55 time=946 ms
    64 bytes from xxx                          : icmp_seq=271 ttl=55 time=1023 ms
    64 bytes from xxx                          : icmp_seq=272 ttl=55 time=928 ms
    64 bytes from xxx                          : icmp_seq=273 ttl=55 time=945 ms
    64 bytes from xxx                          : icmp_seq=274 ttl=55 time=1049 ms
    64 bytes from xxx                          : icmp_seq=275 ttl=55 time=1014 ms
    64 bytes from xxx                          : icmp_seq=276 ttl=55 time=11.8 ms
    64 bytes from xxx                          : icmp_seq=277 ttl=55 time=11.8 ms
    64 bytes from xxx                          : icmp_seq=278 ttl=55 time=11.7 ms
    64 bytes from xxx                          : icmp_seq=279 ttl=55 time=11.7 ms
    64 bytes from xxx                          : icmp_seq=280 ttl=55 time=11.6 ms
    64 bytes from xxx                          : icmp_seq=281 ttl=55 time=11.6 ms
    64 bytes from xxx                          : icmp_seq=282 ttl=55 time=11.6 ms
    64 bytes from xxx                          : icmp_seq=283 ttl=55 time=11.7 ms