Custom RSS for ENIs-阿里云帮助中心

Receive Side Scaling (RSS) for an Elastic Network Interface (ENI) uses a hash algorithm to distribute network traffic across multiple CPU cores. This balances loads, improves performance, and prevents single-core overload. RSS is useful in high-concurrency scenarios, such as cloud computing, video stream distribution, and high-frequency trading systems. If you handle special protocols, such as those for encrypted communication or virtual networks, you can customize RSS rules based on your service features. This resolves uneven traffic distribution and maximizes the use of multi-core resources.

What is RSS

Receive Side Scaling (RSS) is an intelligent traffic scheduling mechanism based on a hash algorithm. Its core objective is to use the ENI hardware and driver to calculate a hash value based on packet features, such as source and destination IP addresses and ports. This process maps different network traffic streams to multiple RX queues. The bound CPU cores then process the queues in parallel. This achieves load balancing, increases throughput, and reduces single-core processing latency in a multi-core CPU architecture.

An ENI that supports multiple queues achieves parallel processing using independent RX (receive) and TX (send) channels for each queue. The RX queues are responsible for receiving and staging data packets from the network. When a data packet arrives at the ENI, the ENI hardware uses an RSS policy, such as polling or stream-based allocation, to dynamically distribute different data streams to multiple RX queues, which splits the traffic. By default, the RSS policy for an Elastic Compute Service (ECS) instance is a fixed configuration in the VPC and you cannot view or modify it from within the instance.

Some instance families support the custom RSS feature for ENIs. After you enable the custom RSS feature on a supported instance family, the ENI distributes traffic based on the default custom hashing rules. You can also further adjust the RSS configuration based on actual traffic characteristics.

After you enable custom RSS on the network interface card, you can also configure RSS in DPDK scenarios to maximize multi-core performance.

Principles and mechanisms

The core implementation of custom RSS for an ECS instance ENI is as follows:

Hash value calculation: The ENI calculates a hash value based on the 5-tuple (source IP, destination IP, source port, destination port, and protocol) of a received packet, a hash key, and a hash algorithm.
- Hash key: The hash key has a fixed size of 40 bytes. The network interface card driver of an ECS instance generates a default hash key when it initializes the network interface card. You can adjust the hash key as needed to affect traffic distribution.
- Hash algorithm: Currently, only the default toeplitz algorithm is supported. This setting cannot be modified.
- Hashing rules: The rules for calculating hash values vary by traffic type. The default hashing rules for ECS ENI RSS are as follows. You cannot change these rules.
  - TCP and UDP traffic over IPv4/IPv6: A hash value is calculated based on the 4-tuple (source IP, destination IP, source port, and destination port) and the hash key.
  - Non-TCP/UDP traffic over IPv4/IPv6, such as ICMP: A hash value is calculated based on the 2-tuple (source IP and destination IP) and the hash key.
  - Non-IP messages (such as ARP) are sent to the default queue (queue 0) and are not hashed.
Indirection table mapping: The RSS indirection table is a predefined array that maps hash values to destination receive queues. Each element, or hash bucket, stores a destination queue number, such as 0, 1, or 2. This indicates which queue should process the packet. When the ENI driver for an ECS instance loads, it automatically generates an indirection table that evenly distributes traffic.
- Indirection table length: The total number of hash buckets. The value is fixed at 128.
- Distribution modes:
  The distribution mode of the RSS indirection table is the core mechanism that maps hash values to specific receive queues. Using a predefined indirection table, the result of the hash calculation is converted into a queue index. This allows for flexible traffic distribution across multiple cores.
  Different distribution modes, such as uniform distribution and weighted distribution, directly affect how traffic is distributed across CPU cores. This in turn affects system throughput, latency, resource utilization, and service performance.
  You can choose a suitable distribution pattern based on your specific application scenario. For more information, see Indirection table distribution pattern.
- Indirection table mapping process:
  1. Hash bucket index calculation: Index = Hash value % Indirection table length (For example, 88 % 128 → index 88). You can use the RSS calculation script to calculate the index value.
  2. Fill queue numbers: You can write the queue numbers into each hash bucket of the indirection table based on the distribution mode.
    After the indirection table is generated, you can find the corresponding queue number based on the calculated index. The packet is then sent to that queue for processing.
CPU core binding: When a queue receives a packet, it triggers a corresponding interrupt. The bound CPU core then processes the interrupt.
Images other than Red Hat Enterprise Linux support network interrupt affinity by default. This means that each queue is associated with an independent interrupt, and the configured interrupt affinity distributes interrupt handling across different CPU cores. For more information, see Core mechanism of multi-queue.

Enable or disable custom RSS for an ENI

Important

The custom RSS feature for ENIs is in invitational preview. To use this feature, you can submit a ticket.

After you enable the custom RSS feature for an ENI on a supported instance type, received traffic is distributed to multiple receive queues based on the default hashing rules. Different CPU cores process the queues in parallel. This increases throughput and reduces single-core load.

Conditions

The custom RSS feature for ENIs is being rolled out to regions in phases. The feature is available in the following regions:
Region Name
Region ID
China (Hong Kong)
cn-hongkong
Currently, only some instance families that support multi-queue for network interface cards support custom RSS, including the general-purpose instance family g9i and the memory-optimized instance family r9i.
You can call the DescribeInstanceTypes API operation to query the support status. A return value of `true` for RssSupport indicates that the feature is supported and a value of `false` indicates that it is not.
We recommend that you use an Alibaba Cloud Linux 3.2104 LTS 64-bit image.
- If you use other public images, the kernel version must be 6.12 or later.
- If you use the custom RSS feature in a DPDK application, the DPDK version must be 21.11 or later.

How to enable or disable

The custom RSS feature for ENIs is disabled by default. You can enable or disable this feature when you create an ENI or after the ENI is created.

When you call the CreateNetworkInterface operation, you can set the EnableRss parameter in EnhancedNetwork to true or false to create an ENI with the custom RSS feature enabled or disabled.
The custom RSS feature takes effect after the ENI is attached to an instance.
You can also call ModifyNetworkInterfaceAttribute and set the EnableRss parameter in the EnhancedNetwork parameter to true or false to modify the custom RSS feature for the network interface card. After the feature is enabled or disabled, the change takes effect when the network interface card is reattached to an instance:
- For a secondary ENI that is attached to an instance, you must detach the ENI and then attach the ENI for the change to take effect.
- For the primary network interface, you need to restart the instance for the change to take effect.
You can call DescribeNetworkInterfaceAttribute and set Attribute to enhancedNetwork to query whether custom RSS is enabled for a network interface. A returned EnableRSS value of true indicates that custom RSS is enabled, and a value of false indicates that it is disabled.
If the RSS feature has not been modified for the ENI, this operation does not return the EnableRSS parameter, which means that the feature is not enabled.

View the RSS configuration of an ENI

After you enable custom RSS for a network interface card, the network interface card driver generates a default RSS configuration according to the default mechanism and rules when the network interface card is reloaded.

You can remotely log on to a Linux instance and run ethtool -x eth0 to view the RSS configuration of the primary network interface card. For a secondary ENI, you can modify the network interface card identifier, such as eth1 or eth2.

Hash table (RX flow hash indirection table): Defines the mapping rules from hash values to the 64 receive queues.
- Each row in the hash table represents a range of hash values, indicated by the starting index. For example, 0 indicates hash values 0-7, and 8 indicates 8-15.
- The numbers in the table (0, 1, 2, and so on) represent the IDs of the corresponding receive queues. In this example, there are 64 queues (0-63).
- Current configuration: The indirection table is cyclically filled with the sequence 0,1,2,...,63,0,1,2...63, which ensures that hash values are evenly mapped to the 64 queues and that traffic is evenly distributed.
RSS hash key: The key used to calculate the hash value of a packet. It affects the uniformity of traffic distribution. The hexadecimal characters in the example represent a 40-byte key used for hash calculation.
RSS hash function: Defines the algorithm for calculating the hash value.
- toeplitz: This is the default algorithm and is currently enabled. It supports symmetric hashing based on a 5-tuple and is suitable for general-purpose traffic.
- xor/crc32: These are other optional algorithms that are typically used in specific scenarios. ECS currently supports only the toeplitz algorithm, and other algorithms are not supported or enabled.
Hashing rules: You can run the following command to view the hash field configuration for the traffic received by the primary ENI.
```
ethtool  -n eth0 rx-flow-hash tcp4
```
- You can replace tcp4 with the protocol type that you want to query, such as udp4, tcp6, or udp6.
- The default hashing rules for ECS ENI RSS are as follows. You cannot change these rules.
  - TCP and UDP traffic over IPv4/IPv6: A hash value is calculated based on the 4-tuple (source IP, destination IP, source port, and destination port) and the hash key.
  - Non-TCP/UDP traffic over IPv4/IPv6, such as ICMP: A hash value is calculated based on the 2-tuple (source IP and destination IP) and the hash key.
  - Non-IP messages (such as ARP) are sent to the default queue (queue 0) and are not hashed.
- Taking tcp4 as an example, for IPv4 TCP traffic, the hash value is calculated by default based on the 4-tuple (source IP, destination IP address, source port, destination port) and the hash key:
  The expected traffic features are as follows:
  - Same TCP connection: All packets have the same hash value and are distributed to the same queue. This ensures packet ordering.
  - Different TCP connections: If the 4-tuples are different, the hash values are different. The traffic is distributed to different queues. This allows multiple CPU cores to process different connections in parallel and increases throughput.

If the following message is returned, it indicates that custom RSS is not enabled for the network interface card or that the instance type to which the card is attached does not support custom RSS. You need to enable the custom RSS feature for the network interface card after you confirm that the instance type supports it.

Configure custom RSS for an ENI

After you enable custom RSS for an ECS instance's ENI, the default RSS configuration can meet the requirements of most scenarios. However, you may need to optimize and adjust the configuration by changing the hash key or configuring the indirection table in the following situations:

Uneven traffic: The default hash values may cause specific traffic to be concentrated in a few queues.
Performance optimization: You can adjust the hashing rules based on service traffic features, such as a high proportion of UDP traffic, to optimize queue utilization.
Security requirements: You can use a custom hash key to prevent potential attackers from predicting traffic distribution paths.

Important

In this example, the instance is configured with the Alibaba Cloud Linux 3.2104 LTS 64-bit image.
This example uses the primary ENI eth0 to describe the steps for manually setting RSS. You can replace the network interface identifier with another one, such as eth1 or eth2, to modify the RSS configuration for the corresponding secondary ENI.
When the RSS configuration of an ENI does not match the features of your service traffic, it can cause issues such as uneven traffic distribution and cross-core state contention, which can negatively affect network performance and application stability. You must adjust the configuration based on the actual features of your service traffic and application requirements. After making adjustments, you can use monitoring tools to observe the data packets received by each queue in real time and adjust the configuration promptly.

Configure the hash key

You may need to regenerate the RSS hash key if you find that traffic is unevenly distributed (for example, the rx_packets of some queues are significantly higher than others), or to prevent potential attackers from inferring traffic patterns by reverse-engineering the hash value.

Important

Modifying the key changes the hash values of existing connections. This may cause temporary packet reordering or retransmissions. We recommend that you perform this operation during off-peak hours.

Run the following command to view the current hash key of the ENI.
The ENI driver for an ECS instance generates a 40-byte default hash key when the ENI is loaded during initialization.
```
ethtool -x eth0
```
Run the following command to generate a new random key using OpenSSL.
```
openssl rand -hex 40 | fold -w2 | paste -sd: -
```
Run the following command to apply the new key to the ENI.
Important
This is a temporary setting. It becomes invalid after the instance is restarted or the ENI is reattached. The ENI driver automatically initializes a random default configuration.
```
ethtool -X eth0 hkey <hash key>
```
Replace <hash key> with the new key that you generated in the previous step.
Run the following command to verify that the new key has taken effect.
```
ethtool -x eth0
```
The output shows that the new key has taken effect:

Configure the indirection table

The distribution mode of the RSS indirection table is the core mechanism that maps hash values to specific receive queues. Using a predefined indirection table, the result of the hash calculation is converted into a queue index. This allows for flexible traffic distribution across multiple cores.

Different distribution modes, such as uniform distribution and weighted distribution, directly affect how traffic is distributed across CPU cores. This in turn affects system throughput, latency, resource utilization, and service performance.

Different distribution modes are suitable for different business scenarios. You can adjust the configuration as needed.

Important

The following example configurations are temporary. They become invalid after the instance is restarted or the ENI is reattached. The ENI driver automatically initializes a random default configuration.

Uniform distribution mode: You can configure the RSS indirection table to use the uniform distribution mode. The first N queues are used to fill the hash buckets in a loop. This mode is suitable for general high-concurrency business scenarios.
```
ethtool -X eth0 equal <Number of queues N>
```
If the value is set to 64, the indirection table is filled in a loop with 0,1,2,...,63,0,1,2...63,.
Weighted distribution: You can allocate hash buckets based on a weight ratio to support asymmetric loads. This mode is suitable for scenarios with differentiated business priorities (for example, queue 0 processes real-time traffic and queue 1 processes background tasks) or mixed CPU performance.
```
ethtool -X eth0 weight <queue 0 weight> <queue 1 weight> ...
```
In the following example, the weight for queue 0 is 60%, and the weight for queue 1 is 40%:
Note
If there are four queues in total, but you only set weights for two queues, the indirection table will only generate mappings for queue 0 and queue 1.
```
ethtool -X eth0 weight 6 4
```
Partial queue distribution: Starting from a specified queue, you can use a consecutive number of queues to fill the hash buckets in a loop. This mode is suitable for scenarios where you want to direct traffic to a specific CPU range, such as a NUMA node.
```
ethtool -X eth0 start <start queue> equal <number of queues>
```
In the following example, 40 queues starting from queue 2 (queues 2-41) are used to fill the hash buckets in a loop:
```
ethtool -X eth0 start 2 equal 40
```

Use `watch` to observe traffic distribution

After you enable the network interface card for the instance and configure custom RSS, you can use hping3 to generate traffic with different features and use watch to monitor the interrupt distribution of the network interface card queues to verify whether traffic is distributed to multiple cores as expected under the current RSS configuration.

Preparations

You can purchase two ECS instances with the following configurations:

Sending ECS instance (10.0.0.252): An instance installed with hping3 to generate traffic with different features.
Accepter ECS instance: An instance whose type supports multi-queue for ENIs, to which a secondary ENI is attached (10.0.0.5), and for which custom RSS is enabled for the secondary ENI eth1.
Note
- This example uses four queues for testing.
- To eliminate the impact of traffic from the remote SSH logon to the acceptor instance (primary ENI), we enable custom RSS on the secondary ENI eth1 for testing.
Network connectivity: The two ECS instances are in the same security group and can communicate with each other over the internal network.

Procedure

You can remotely log on to the accepter ECS instance, check the RSS configuration of the secondary ENI, and confirm the expected traffic distribution. For more information, see View the RSS configuration of an ENI.
You can remotely log on to the sender ECS instance and run the following command to install hping3.
```
yum install -y hping3
```
On the acceptor ECS instance, run the following command to monitor the queue packet counts in real time.
```
watch -n 1 "ethtool -S eth1 | grep rx[0,1,2,3]_packets"
```
Replace the queue number configuration with the actual configuration of the acceptor ENI.
You can log on remotely to the sender ECS instance and run the following commands to simulate different traffic patterns.
- Scenario 1: Send 10,000 SYN packets at high speed to the acceptor IP address with random destination ports. This ensures that the traffic hash is dispersed and verifies hash balancing.
```
sudo hping3 10.0.0.5 -S -a 10.0.0.252 --rand-dest -p 0 --baseport 10000 -c 10000 -i u100 -I eth0
```
  - rand-dest: Random destination port
  - -p 0: Used with rand-dest
  - --baseport 10000: The starting value for the source port.
  - -c 10000: Send 10,000 packets
  - -i u100: Specifies an interval of 100 microseconds between packets for high-speed sending.
  - -I eth0: Used with rand-dest to specify the sender's network interface
  Observe the output on the acceptor ECS instance. The packet counts of the four queues are expected to increase in a balanced manner:
- Scenario 2: Send traffic with a fixed source IP address and port. This tests whether the same traffic is mapped to the same queue and verifies hash consistency.
  The following command sends 10,000 TCP SYN packets to the target 10.0.0.5:80, with the source IP fixed to 10.0.0.252 and the source port fixed to 12345.
```
sudo hping3 10.0.0.5 -S -p 80 -c 10000 -s 12345 -a 10.0.0.252 --keep -i u100
```
  - The hash index value is calculated based on the RSS script. The traffic is expected to be processed by queue 0:
  - Actual observation result on the acceptor ECS instance (10.0.0.5):
If the service traffic distribution does not meet your expectations, for example, when a queue has a significantly high load or the load is unbalanced, you can try to optimize the distribution by adjusting the hash key or configuring an indirection table.

Configure and use RSS in DPDK

The Data Plane Development Kit (DPDK) is an open-source user-mode data plane acceleration framework initiated by Intel and now maintained by the Linux Foundation. It is designed to optimize network packet processing performance. DPDK uses technologies such as user-mode drivers, zero-copy, and polling mode to increase network packet processing performance to near line speed, making it a cornerstone for building high-performance network applications. It is suitable for domains sensitive to throughput and latency, such as telecom clouds, fintech, and edge computing. For more information about DPDK, see Data Plane Development Kit (DPDK*).

Receive Side Scaling (RSS) is a hardware feature of an ENI that distributes traffic to different queues for processing by different CPU cores. This is critical for the multi-core processing model of the DPDK. After you enable custom RSS on the ENI, you can use testpmd and l3fwd in DPDK to test and verify the RSS feature.

Install and configure DPDK on an ECS instance

Note

If you use the custom RSS feature in a DPDK application, the DPDK version must be 21.11 or later.
This topic uses an ecs.r9i.16xlarge instance (64 queues) running the Alibaba Cloud Linux 3.2104 LTS 64-bit image as an example to demonstrate the installation of DPDK version 22.11.3.
This example uses the primary ENI eth0 to describe the steps for manually setting RSS. You can replace the network interface identifier with another one, such as eth1 or eth2, to modify the RSS configuration for the corresponding secondary ENI.

Step 1: Install DPDK

Click to view the example steps to install DPDK

You can update the system and install basic tools.

sudo yum update -y
sudo yum install -y git wget gcc make kernel-devel-$(uname -r) numactl-devel python3 pciutils

You can install DPDK dependencies.

sudo yum install -y libpcap-devel meson ninja-build

You can configure enormous pages.
- DPDK bypasses the kernel protocol stack to directly operate the ENI. It needs to efficiently manage memory to reduce the Translation Lookaside Buffer (TLB) miss rate. Enormous pages use larger memory pages (usually 2 MB) than traditional 4 KB pages. This reduces the number of TLB misses during address translation, thereby improving memory access speed.
- If too many enormous pages are allocated, it reduces the normal memory space available for the operating system to dynamically allocate. This may cause other non-huge-page applications or system services to run poorly or fail due to insufficient memory. If the enormous page setting for an instance is too large, it may cause issues such as being unable to connect to the instance.
- You can calculate the required number of enormous pages based on your application's actual memory needs and the total system memory.
  Number of enormous pages = (Application's required memory size / Actual size of a enormous page). In Linux, the default enormous page size is usually 2 MB. For example, if an application requires 16 GB of enormous page memory and the enormous page size is 2 MB, the required number of enormous pages is 16 GB / 2 MB = 8192.
```
echo "vm.nr_hugepages = 8192" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
```

You can create and mount the enormous pages directory.

sudo mkdir -p /dev/hugepages
sudo mount -t hugetlbfs hugetlbfs /dev/hugepages

You can download the DPDK source code. An Internet connection is required.

cd ~
wget https://fast.dpdk.org/rel/dpdk-22.11.3.tar.xz
tar xf dpdk-22.11.3.tar.xz
cd dpdk-stable-22.11.3

You can compile DPDK.

# Initialize the build directory and configure project options, specifying to build the l3fwd Layer 3 forwarding example.
meson setup -Dexamples=l3fwd build
cd build
# Compile.
ninja
# Install the compiled files to the system directory.
sudo ninja install
# Update the system's shared library cache.
sudo ldconfig

If you encounter the missing python module: elftools error shown in the figure below during compilation,

you need to specify your current Python version, install the pyelftools library, and then compile again:

sudo /usr/bin/python3.8 -m pip install pyelftools

Step 2: Load kernel modules

DPDK requires kernel modules such as UIO or VFIO to support user-mode device access. VFIO is generally preferred for its enhanced security (relies on IOMMU), while UIO is suitable for quick testing. This topic uses the VFIO driver as an example.

You can enable IOMMU.
VFIO relies on IOMMU for secure user-mode device binding and DMA mapping, so IOMMU must be enabled.
1. Run the following command to open the GRUB configuration file:
```
sudo vim /etc/default/grub
```
2. Press the I key to enter Insert mode, add intel_iommu=on to the end of the line for GRUB_CMDLINE_LINUX, and then save the configuration file.
  The following figure shows an example of the modified configuration file.
3. Run the following command to apply the modified configurations:
```
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
```
4. Run the following command to restart the instance. Then, reconnect to the instance.
```
reboot
```
  Warning
  The restart operation stops the instance for a short period of time and may interrupt services that are running on the instance. We recommend that you restart the instance during off-peak hours.
Run the following commands to install the VFIO and VFIO-PCI drivers.
```
sudo modprobe vfio && \
sudo modprobe vfio-pci
```

Run the following command to enable no-IOMMU mode:

sudo bash -c 'echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode'

Step 3: Bind the ENI to the DPDK driver

Enable custom RSS on the network interface card.
Make sure that custom RSS is enabled for the ENI that the DPDK driver will take over, and that the ENI has been reattached to the instance for the change to take effect. The ENI driver will generate a default RSS configuration when it reloads.
Connect to an instance using VNC.
- This example uses the primary ENI. The subsequent operations involve detaching the ENI, which interrupts an SSH session.
- If you are performing an operation on the secondary ENI, you can still connect to the instance using Workbench (primary ENI) to perform the configuration.
View the PCI device driver binding status of the ENI. By default, the ENI is being used by the system kernel.
```
dpdk-devbind.py --status
```
- The device 0000:00:05.0 (PCI device identifier) is currently managed by the virtio-pci kernel driver, and its corresponding network interface eth0 is active (has an IP address configured or is transmitting data).
- The device can be switched to the vfio-pci driver (for DPDK user-mode takeover), but you must first disable the interface and detach the kernel driver.
Deactivate eth0.
```
sudo ip link set dev eth0 down
```
If you do not deactivate it, an error will be reported when binding to VFIO:
Unbind the kernel driver and bind to VFIO.
```
dpdk-devbind.py -b vfio-pci 0000:00:05.0
```
Replace the PCI device identifier with the one that you queried for your ENI.
Note
You can stop the DPDK application by running sudo pkill dpdk-app and then run dpdk-devbind.py -b virtio-pci 0000:00:05.0 to rebind the kernel driver.
Run the following command again to view the PCI device driver binding status of the ENI. The ENI is expected to be taken over by DPDK.
Important
When the network interface card is taken over by the DPDK user mode driver, the kernel no longer controls the device. Therefore, you cannot use commands such as ip a to view its information.
```
dpdk-devbind.py --status
```
You can see that the device 0000:00:05.0 is currently bound to the user mode driver vfio-pci.

Configure RSS using testpmd

Testpmd (Test Packet Mode Driver) is a core testing tool in DPDK. It is mainly used to quickly verify DPDK features, test the performance of ENI drivers, and debug data plane applications. For more information about using testpmd, see Testpmd Runtime Functions.

Important

Modifications to the ENI configuration, such as the number of queues, RSS rules, and the RETA table, take effect only after you start or restart packet forwarding.

Connect to an instance using VNC.
Start the DPDK testpmd packet forwarding test tool.
```
dpdk-testpmd -a 0000:00:05.0 --socket-mem 1024 -- -i --portmask=0x1 --rxq=64 --txq=64  --forward-mode=rxonly
```
- -a 0000:00:05.0: Attaches the network interface card with the PCI address 0000:00:05.0 to DPDK. You can run dpdk-devbind.py --status to view the PCI address.
- --socket-mem 1024: Pre-allocates 1024 MB of enormous page memory for each NUMA node.
- -i: Starts the application in interactive command-line mode. This mode lets you dynamically adjust the configuration, such as modifying the forwarding pattern or viewing statistics information. You can enter quit to exit.
- --portmask=0x1: Enables port mask 0x1 (binary 0001), which indicates that only the first network interface card is used (that is, the network interface card at PCI address 0000:00:05.0).
  - In DPDK, each binary bit of portmask corresponds to a port number (for example, 0x1 enables port 0, and 0x3 enables ports 0 and 1).
  - In this example, only one device (0000:00:05.0) is attached to DPDK, so its port number in DPDK is 0.
- --rxq=64 / --txq=64: Sets the number of receive queues (RXQs) and send queues (TXQs) for each port to 64 to enable multi-queue parallel processing. You can replace 64 with the actual number of queues for the Elastic Network Interface (ENI).
- --forward-mode=rxonly: Specifies the forwarding mode as receive-only (packets are received and then discarded without being sent). This mode is used to test the receiving performance of a network interface card or for packet capture.
You can enter interactive mode and query the current RSS configuration of the ENI.
- Query hash configuration information: show port info <port_id>
  port_id: Specifies the port. In this example, it is port 0.
  Click to view example output
- Query hash key: show port <port_id> rss-hash key
- Query indirection table configuration: show port <port_id> rss reta <size> <mask0, mask1...>
  - size: The number of indirection table entries to query. The value is fixed at 128.
  - mask0, mask1: The masks used to filter the range of hash indexes to be displayed, which are specified in hexadecimal format. For example, mask0=0xff indicates that entries for hash indexes 0 to 7 are displayed.
    The indirection table size is fixed at 128. Two masks are required, each covering 64 index blocks. Running the command show port 0 rss reta 128 (0xffffffffffffffff,0xffffffffffffffff) returns the mapping information for all 128 index blocks and queues:
    Click to view example output
You can configure RSS based on your requirements.
- Configure a newly generated hash key.
  You can use OpenSSL to generate a new random key.
```
port config <port_id> rss-hash-key (ipv4|ipv4-frag|\
                  ipv4-tcp|ipv4-udp|ipv4-sctp|ipv4-other|\
                  ipv6|ipv6-frag|ipv6-tcp|ipv6-udp|ipv6-sctp|\
                  ipv6-other|l2-payload|ipv6-ex|ipv6-tcp-ex|\
                  ipv6-udp-ex <string of hex digits \
                  (variable length, NIC dependent)>)
```
  For TCP over IPv4, run the port config 0 rss-hash-key ipv4 6D5A56DA255B0EC24167253D43A38FB0D0CA2BCBAE7B30B477CB2DA38030F20C6A42B73BBEAC01FC command to configure the hash key and ensure that the configuration takes effect:
- Configure the RSS indirection table to map specific hash values to specified queues.
```
port config all rss reta <hash,queue>,<hash,queue>..
```
  You need to configure this based on the actual number of queues:
  - hash: The hash index. The range depends on the size of the indirection table (for example, 0-63 for a 64-entry table).
  - queue: The number of the target receive queue, such as queue 0 or queue 1.

Apply RSS in l3fwd

To enable RSS in a DPDK application, you must implement the configuration for the RSS hash key, indirection table, and other settings in the relevant code. The following section uses L3FWD as an example.

L3FWD (Layer 3 Forwarding) is a Layer 3 network forwarding sample application in DPDK. It demonstrates how to implement high-performance packet routing and forwarding based on IP addresses. It uses DPDK technologies such as zero-copy and Polling Mode Drivers (PMDs) to achieve near-line-speed Layer 3 forwarding performance.

You can modify the L3FWD source code in DPDK (examples/l3fwd/main.c).

Modify the port initialization section static struct rte_eth_conf port_conf in the L3FWD sample code.

Click to view the code modification

#define RSS_HASH_KEY_LENGTH 	40
#define RSS_RETA_SIZE 			128

static uint8_t hash_key[RSS_HASH_KEY_LENGTH] = {
    0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A,
    0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A,
    0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A,
    0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A,
};

static struct rte_eth_conf port_conf = {
	.rxmode = {
		.mq_mode = RTE_ETH_MQ_RX_RSS,
		.offloads = RTE_ETH_RX_OFFLOAD_UDP_CKSUM | RTE_ETH_RX_OFFLOAD_TCP_CKSUM,
	},
	.rx_adv_conf = {
		.rss_conf = {
			.rss_key = hash_key,
			.rss_hf = RTE_ETH_RSS_IP,
			.rss_key_len = RSS_HASH_KEY_LENGTH,
		},
	},
	.txmode = {
		.mq_mode = RTE_ETH_MQ_TX_NONE,
	},
};

Add functions to configure the hash indirection table and the hash key.

Click to view the added code

/**
 * @brief Configure the RSS RETA (Redirection Table Array) for a given port.
 * 
 * @param port_id The ID of the port to configure.
 */

void configure_rss_reta(uint16_t port_id) {
    struct rte_eth_rss_reta_entry64 reta_conf[RSS_RETA_SIZE / RTE_ETH_RETA_GROUP_SIZE];
    unsigned int i;
    uint16_t reta_size;
	uint16_t nb_queues;
	struct rte_eth_dev_info dev_info;

	if (port_id >= RTE_MAX_ETHPORTS) {
        printf("port_id %d exceed max eth ports\n", port_id);
        return;
    }

    if (rte_eth_dev_info_get(port_id, &dev_info) != 0) {
		printf("Failed to get device info for port %d\n", port_id);
		return;
	}

    reta_size = dev_info.reta_size;
    if (reta_size == 0) {
        printf("Device does not support RSS RETA configuration.\n");
        return;
    }

	nb_queues = dev_info.nb_rx_queues;
	if (nb_queues == 0) {
		printf("port %d RX queues = 0\n", port_id);
		return;
	}

    // Initialize RETA table
    memset(reta_conf, 0, sizeof(reta_conf));
    for (i = 0; i < reta_size; i++) {
        reta_conf[i / RTE_ETH_RETA_GROUP_SIZE].reta[i % RTE_ETH_RETA_GROUP_SIZE] = (uint16_t)(i % nb_queues);
    }

    // Configure RETA table mask
    for (i = 0; i < reta_size; i += RTE_ETH_RETA_GROUP_SIZE) {
        reta_conf[i / RTE_ETH_RETA_GROUP_SIZE].mask = UINT64_MAX;
    }

	// Update RSS RETA table to device
	if (rte_eth_dev_rss_reta_update(port_id, reta_conf, reta_size) != 0) {
		printf("Failed to update RSS RETA table for port %u\n", port_id);
		return;
	}
}

/**
 * @brief Configure the RSS hash key for a given port.
 * 
 * @param port_id The ID of the port to configure.
 */
void configure_rss_hash_key(uint16_t port_id) {
    struct rte_eth_rss_conf rss_conf;

	if (port_id >= RTE_MAX_ETHPORTS) {
        printf("port_id %d exceed max eth ports\n", port_id);
        return;
    }

    memset(&rss_conf, 0, sizeof(rss_conf));
    rss_conf.rss_key = hash_key;
    rss_conf.rss_key_len = sizeof(hash_key);
    
    // Update RSS hash key to device
    if (rte_eth_dev_rss_hash_update(port_id, &rss_conf) != 0) {
        printf("Failed to update RSS hash key for port %u\n", port_id);
		return;
    }
}

After rte_eth_dev_start, call the new configuration functions.

Click to view the calling code in main

@@ -1472,12 +1576,15 @@ main(int argc, char **argv)
                /* Start device */
		ret = rte_eth_dev_start(portid);
		if (ret < 0)
			rte_exit(EXIT_FAILURE,
				"rte_eth_dev_start: err=%d, port=%d\n",
				ret, portid);
				
		configure_rss_reta(portid);
		configure_rss_hash_key(portid);

After you modify the source code, you can recompile L3FWD.

cd ~/dpdk-stable-22.11.3/
rm -rf build
# Initialize the build directory and configure project options, specifying to build the l3fwd Layer 3 forwarding example.
meson setup -Dexamples=l3fwd build
cd build
# Compile.
ninja
# Install the compiled files to the system directory.
sudo ninja install
# Update the system's shared library cache.
sudo ldconfig

Specify the bindings between ports, queues, and cores and start L3FWD.
```
cd ~/dpdk-stable-22.11.3/build/examples
./dpdk-l3fwd --legacy-mem -a 0000:00:05.0 --socket-mem 1024 -- -p 0x1 --config="(PORT_ID, QUEUE_ID, LCORE_ID), (PORT_ID, QUEUE_ID, LCORE_ID), ..." --parse-ptype
```
- --config: Each trituple represents the following:
  - PORT_ID: The ID of the attached port. The value starts from 0.
  - QUEUE_ID: The ID of the receive queue for the port. The value starts from 0.
  - LCORE_ID: The ID of the logical core (lcore) that processes the queue. The value starts from 0.
- The following example uses two cores and two queues:
```
./dpdk-l3fwd --legacy-mem -a 0000:00:05.0 --socket-mem 1024 -- -p 0x1 --config="(0,0,0),(0,1,1)"  --parse-ptype
```
  - The first trituple (0,0,0): Logical core 0 (lcore0) processes queue 0 of port 0.
  - The second trituple (0,1,1): Logical core 1 (lcore1) processes queue 1 of port 0.

Use the RSS script to calculate a hash index

After the RSS rules are configured, you can use the provided Python script to calculate the receive queue by entering information such as the 4-tuple and hash key. This script calculates the index value in the indirection table that the ECS backend computes for a specific 4-tuple packet with a specified hash key. This can guide the configuration of the indirection table to receive specific packet streams on a designated queue.

Click to view ali_ecs_rss_calc.py script content

#!/usr/bin/python

import sys
import argparse
import re
import ipaddress

prog_name = sys.argv[0]
USAGE_EXAMPLE = """
Usage example:
    Calculate the Toeplitz hash of a packet sent from 1.2.3.4 to 1.2.3.5 with a source and destination
    port of 7000:

    - The hash key argument is required because the virtio-net driver creates a random hash key for all your NICs.

    $ {prog_name} -t 1.2.3.4 -T 7000 -r 1.2.3.5 -R 7000 -k 77:d1:c9:34:a4:c9:bd:87:6e:35:dd:17:b2:e3:23:9e:39:6d:8a:93:2a:95:b4:72:3a:b3:7f:56:8e:de:b6:01:97:af:3b:2f:3a:70:e7:04
    
    - If you want to calculate the RSS value of a packet whose protocol is not TCP or UDP, do not specify
      the source and destination ports. The script will calculate the hash value based on only the IP addresses.
      
    $ {prog_name} -t 1.2.3.4 -r 1.2.3.5 -k 77:d1:c9:34:a4:c9:bd:87:6e:35:dd:17:b2:e3:23:9e:39:6d:8a:93:2a:95:b4:72:3a:b3:7f:56:8e:de:b6:01:97:af:3b:2f:3a:70:e7:04
    
    - Use "--ipv6" to calculate the RSS value for IPv6 packets.
    
    $ {prog_name} -t 2001:250:250:250:250:250:250:1 -T 7000 -r 2001:250:250:250:250:250:250:2 -R 7000 -k 77:d1:c9:34:a4:c9:bd:87:6e:35:dd:17:b2:e3:23:9e:39:6d:8a:93:2a:95:b4:72:3a:b3:7f:56:8e:de:b6:01:97:af:3b:2f:3a:70:e7:04 --ipv6

    Note: Linux kernel 5.9 or later is required for hash function and key configuration support.
    
    Also, Linux kernels older than ANCK 5.10-018 had a bug where the default hash key shown by ethtool was not 
    the same as the one used by the device before any RSS configuration was manually changed. The script will print the
    correct hash value in this case if you do not specify the hash key during calculation.
""".format(prog_name = prog_name)

# The default key on instances for old alinux kernel(< ANCK 5.10-018) should be as below, not the one you get from "ethtool -x <nic_name>".
RSS_DEFAULT_KEY = [
	0x6D, 0x5A, 0x56, 0xDA, 0x25, 0x5B, 0x0E, 0xC2,
	0x41, 0x67, 0x25, 0x3D, 0x43, 0xA3, 0x8F, 0xB0,
	0xD0, 0xCA, 0x2B, 0xCB, 0xAE, 0x7B, 0x30, 0xB4,
	0x77, 0xCB, 0x2D, 0xA3, 0x80, 0x30, 0xF2, 0x0C,
	0x6A, 0x42, 0xB7, 0x3B, 0xBE, 0xAC, 0x01, 0xFA,
]
# The default key on instances for new alinux kernel(>= ANCK 5.10-018) is randomly generated.

TOEPLITZ_KEY_SIZE = 128
BITS_IN_BYTE = 8

def circular_shift_key_one_left(key):
    """Performs a cyclic left shift of the entire key.
    To cyclically shift all 40 bytes, the function
    shifts bits between adjacent bytes one at a time."""

    l = len(key)
    return [ ((key[i] << 1) & 0xff) | ((key[(i + 1) % l] & 0x80) >> 7) for i in range(0, l) ]

def or_32msb_bits_of_key(key):
    return (key[0] << 24) | (key[1] << 16) | (key[2] << 8) | key[3]

def calculate_hash(rx_ip, rx_port, tx_ip, tx_port, initial_value, key):
    """Calculates the Toeplitz hash based on the provided parameters.
    Note: This implementation is specific to ENA and may not be
    compatible with the standard Toeplitz implementation."""

    hash_result = initial_value
    input_bytes = list()
    input_bytes += tx_ip + rx_ip + tx_port + rx_port

    for input_byte in input_bytes:
        for i in range(BITS_IN_BYTE):
            # is the (8 - i -1) bit set
            if (input_byte & (1 << (BITS_IN_BYTE - i - 1))):
                hash_result ^= or_32msb_bits_of_key(key)

            key = circular_shift_key_one_left(key)

    return hash_result

def ipv4_addr_type(str):
    """An argparse type function that transforms an
    IPv4 string into a list of integers."""
    if not re.match(r"^([0-9]{1,3}\.){3}[0-9]{1,3}$", str):
        raise argparse.ArgumentTypeError("The IP address must be in the format 1.2.3.4.")

    return [int(octet) for octet in str.split('.')]

def ipv6_addr_type(str):
    """An argparse type function that transforms an
    IPv6 string into a list of integers."""
    
    try:
        ip_str = ipaddress.IPv6Address(unicode(str))
    except ValueError as e:
        raise argparse.ArgumentTypeError("Invalid IPv6 address format: %s" % e)
    
    parts = ip_str.exploded.split(':')
    try:
        bytes_list = [int(part, 16) for part in parts]
        bytes_list = [(byte >> 8, byte & 0xff) for byte in bytes_list]
    except ValueError as e:
        raise argparse.ArgumentTypeError("Invalid IPv6 address format: %s" % e)
    
    return [item for sublist in bytes_list for item in sublist]

def toeplitz_key_type(str):
    """An argparse type function that transforms a
    Toeplitz key string into a list of hexadecimal values."""
    if not re.match(r"^([0-9a-zA-Z]{1,2}:){39}[0-9a-zA-Z]{1,2}$", str):
        raise argparse.ArgumentTypeError("The Toeplitz key format is invalid. It must be 40 hexadecimal values delimited by colons.")

    return [int(key_elem, 16) for key_elem in str.split(':')]

def main():

    parser = argparse.ArgumentParser(description='virtio-net Toeplitz hash calculator',
                                     formatter_class=argparse.RawDescriptionHelpFormatter,
                                     epilog=USAGE_EXAMPLE)

    parser.add_argument('-r', '--rx-ip', help='Receiving side IP', dest='rx_ip', nargs='?',
                        required=True, type=str)
    parser.add_argument('-R', '--rx-port', help='Receiving side port', dest='rx_port', nargs='?', type=int)
    parser.add_argument('-t', '--tx-ip', help='Transmitting side IP', dest='tx_ip', nargs='?', 
                        required=True, type=str)
    parser.add_argument('-T', '--tx-port', help='Transmitting side port', dest='tx_port', nargs='?', type=int)
    parser.add_argument('-k', '--toeplitz-key',
                        help='The Toeplitz key (only on instances that support changing it)',
                        dest='toeplitz_key', nargs='?', required=True, type=toeplitz_key_type)
    parser.add_argument('-i', '--ipv6',  action='store_true', help='Use IPv6 address type for IP arguments')

    args = parser.parse_args()

    if args.ipv6:
        rx_ip   = ipv6_addr_type(args.rx_ip)
        tx_ip   = ipv6_addr_type(args.tx_ip)
    else:
        rx_ip   = ipv4_addr_type(args.rx_ip)
        tx_ip   = ipv4_addr_type(args.tx_ip)
    
    if args.rx_port and args.tx_port:
        # "break" port number into two byte representation
        rx_port = [(args.rx_port & 0xff00) >> 8, args.rx_port & 0x00ff]
        tx_port = [(args.tx_port & 0xff00) >> 8, args.tx_port & 0x00ff]
    else:
        rx_port = tx_port = []
    
    key = args.toeplitz_key

    # calculate the hash with an initial value of 0
    hash = calculate_hash(rx_ip, rx_port, tx_ip, tx_port, 0, key)
    rss_table_entry_128 = hash % 128
    rss_table_entry_256 = hash % 256

    if args.ipv6:
        print("Sending traffic from [{}]:{} to [{}]:{}".format(args.tx_ip, args.tx_port, args.rx_ip, args.rx_port))
    else:
        print("Sending traffic from {}:{} to {}:{}".format(args.tx_ip, args.tx_port, args.rx_ip, args.rx_port))
    print("""The hash is calculated over the following fields:
    Source IP address
    Destination IP address""")
    if args.rx_port and args.tx_port:
        print("""    Source port
    Destination port""")
    print("Should result in the hash for all drivers:".ljust(50) + "{}".format(hex(hash)))
    print("RSS table entry (total length 128):".ljust(50) + "{}".format(rss_table_entry_128))
    print("RSS table entry (total length 256):".ljust(50) + "{}".format(rss_table_entry_256))
    return

if __name__ == '__main__':
    main()

Click to view script usage

You can view the script usage by running python ali_ecs_rss_calc.py -h.

The following example uses the RSS configuration of an ENI with 64 queues:

You can calculate the RSS hash value using the RSS calculation script based on the following 5-tuple information. You can modify the values as needed.

Destination IP address (-r): 10.0.0.1, which is the IP address of the acceptor instance's ENI with RSS configured.
Source IP address (-t): 10.0.0.251, which is the IP address of the sender's ENI.
Destination port (-R): 26000
Source port (-T): 18042
Hash key: You can obtain this from the RSS indirection table configuration.

python ali_ecs_rss_calc.py -r 10.0.0.1 -t 10.0.0.251 -R 26000 -T 18042 -k 69:e8:7c:56:bf:03:9f:63:d7:c5:e5:96:b3:00:36:93:02:8c:d2:8f:cc:a9:00:65:fd:c8:94:71:5f:fd:c8:de:7a:30:a9:73:b3:33:0c:c6

The script performs a hash calculation based on the toeplitz algorithm and the specified input parameters. As shown in the example result below, the system supports only an indirection table with a length of 128. A calculated hash index of 117 means that the data packet for this connection is processed by the queue that corresponds to index 117 in the RSS indirection table.

You can view the RSS indirection table configuration again. You can see that the queue corresponding to index 117 is 53. This means that the packets are processed by queue 53.

Region Name	Region ID
China (Hong Kong)	cn-hongkong

What is RSS

Principles and mechanisms

Enable or disable custom RSS for an ENI

Conditions

How to enable or disable

View the RSS configuration of an ENI

Configure custom RSS for an ENI

Configure the hash key

Configure the indirection table

Use watch to observe traffic distribution

Preparations

Procedure

Configure and use RSS in DPDK

Install and configure DPDK on an ECS instance

Step 1: Install DPDK

Step 2: Load kernel modules

Step 3: Bind the ENI to the DPDK driver

Configure RSS using testpmd

Apply RSS in l3fwd

Use the RSS script to calculate a hash index

Use `watch` to observe traffic distribution