On certain older kernel versions, using Bottleneck Bandwidth and Round-trip propagation time (BBR) as the TCP congestion control algorithm without the fq qdisc scheduler causes elevated CPU overhead per connection, degrading network performance for workloads such as Redis databases. This topic explains the cause and provides three solutions.
Affected versions
This issue affects Elastic Compute Service (ECS) instances that meet all of the following conditions:
Image version:
aliyun_2_1903_64_20G_alibase_20190619.vhdor earlierKernel version:
kernel-4.19.48-14.al7or earlierHigh CPU utilization and high packet forwarding rate at the same time
Cause
When BBR is the active TCP congestion control algorithm but the network interface's qdisc does not use the fq scheduler, the TCP stack falls back to its internal high-resolution timer (hrtimer). This increases CPU utilization per connection.
Choose a solution
Select a solution based on your network scenario:
| Scenario | Solution |
|---|---|
| Instance serves traffic only over the internal network | Switch to CUBIC |
| Instance serves traffic over the Internet | Change qdisc to fq |
| Neither option is viable, or you want a permanent fix | Upgrade the kernel |
Switch to CUBIC
CUBIC is well-suited for high-bandwidth, low-latency internal network environments. Switching to CUBIC eliminates the hrtimer overhead caused by BBR without the fq scheduler.
Apply the change immediately.
sudo sysctl -w net.ipv4.tcp_congestion_control=cubicPersist the change across reboots.
echo "net.ipv4.tcp_congestion_control=cubic" | sudo tee -a /etc/sysctl.d/50-aliyun.confVerify the active algorithm.
sysctl net.ipv4.tcp_congestion_controlExpected output:
net.ipv4.tcp_congestion_control = cubic
Change qdisc to fq
For instances that serve Internet-facing traffic, configuring the fq scheduler on the network interface removes the need for BBR's hrtimer fallback, reducing per-connection CPU overhead while keeping BBR active.
The following command applies the change only until the next reboot. To make the configuration persistent, add the command to a startup script or configure the qdisc permanently through your network configuration management tool.
Replace <$DEV> with the actual network interface name (for example, eth0).
sudo tc qdisc add dev <$DEV> root fqVerify that the fq qdisc is active on the interface.
tc qdisc show dev <$DEV>Expected output (exact values may differ):
qdisc fq 8001: root refcnt 2 limit 10000p flow_limit 100p ...Upgrade the kernel
Upgrading to the latest Alibaba Cloud Linux 2 kernel resolves this issue permanently.
Kernel upgrades may cause compatibility and stability issues. Review the release notes for Alibaba Cloud Linux 2 and test in a non-production environment before upgrading.
Restarting the instance temporarily stops it, which may interrupt running services and cause data loss. Back up critical data before proceeding, and schedule the restart during off-peak hours.
Upgrade to the latest kernel version.
sudo yum update kernelRestart the instance for the new kernel to take effect.
sudo rebootAfter the instance restarts, verify the kernel version.
uname -rConfirm the output shows a kernel version later than
kernel-4.19.48-14.al7.