Optimize network interrupt handling with vCPU pinning

Use cases

This optimization is recommended for the following use cases:

• The application is both CPU-intensive and network I/O-heavy, such as Redis, Kafka, Nginx, or a MySQL Proxy.

• The application runs on a multi-core Elastic Compute Service (ECS) instance with 16 or more vCPUs.

• You observe a high percentage of CPU time spent on irq or softirq on the cores where the application process is running. You can check this using mpstat.

This optimization is not recommended for the following scenarios:

• Small instances with fewer than 8 vCPUs.

• Mixed deployments with multiple high-load services.

• Containerized environments, unless CPU resource isolation is configured with tools like the Kubernetes CPU Manager or cgroups.

How it works

Linux network packet processing involves two key stages:

• Hardware Interrupt (Hard IRQ): The Network Interface Card (NIC) receives a packet and triggers a vCPU interrupt that the driver then handles.

• Software Interrupt (Soft IRQ): The kernel protocol stack, such as NET_RX, processes the packet in the context of ksoftirqd or NAPI.

Sharing cores with application threads can cause:

• L1/L2 Cache Pollution

• Frequent flushing of the vCPU pipeline

• Increased scheduling latency

Using IRQ affinity and Receive Packet Steering (RPS) confines interrupt handling to a dedicated set of vCPUs, allowing application threads to exclusively use the remaining cores and achieve resource isolation.

RPS is a software mechanism in the Linux kernel that distributes packets received from a single-queue NIC across multiple vCPU cores for parallel processing during the software interrupt stage. This overcomes single-core bottlenecks and improves network throughput.

Configuration steps

# Get the interrupt numbers for the virtio device corresponding to eth0 (e.g., virtio1)
IRQS=$(grep 'virtio1-' /proc/interrupts | cut -d: -f1 | tr -d ' ')

# Calculate the vCPU mask: vCPUs 8–31 correspond to the hexadecimal mask ffffff00
# For a 32-core system (0–31), this mask sets bits 8 through 31 to 1.
for irq in $IRQS; do
    echo ffffff00 > /proc/irq/$irq/smp_affinity
done

# Set the RPS vCPU mask for each RX queue to match the IRQ mask
for rps_file in /sys/class/net/eth0/queues/rx-*/rps_cpus; do
    echo ffffff00 > "$rps_file"
done

# Pin the Redis process to vCPUs 0-7 on startup
taskset -c 0-7 redis-server /etc/redis.conf

Example: Improving Redis Performance

Deploying Redis 6.2 on a g9i.8xlarge (32 vCPU) instance yielded the following results:

Test command:

./memtier_benchmark -s $SERVER_IP -p $port -t 8 --test-time=60 -c 10 --ratio=0:1 --pipeline=1 -d 32 --key-maximum=100000

Key observation:

• mpstat showed that the %irq and %soft values on application cores dropped to nearly zero.

Verify the optimization

mpstat -P ALL 1

Before pinning:

After pinning: