Elastic RDMA Interface (ERI)
An Elastic RDMA Interface (ERI) is a network interface that you can attach to an ECS instance. It is a key component for implementing elastic Remote Direct Memory Access (eRDMA). An ERI is attached to an Elastic Network Interface (ENI) to enable the RDMA feature on the ENI.
Function introduction
To configure eRDMA on an ECS instance, you must attach an ERI to the instance. eRDMA is an Alibaba Cloud service that provides a high-performance RDMA network that features low latency, high throughput, and high extensibility. Using an ERI, eRDMA enables network communication on ECS instances with ultra-low latency and high throughput. For more information, see eRDMA overview.
Only some instance types support attaching ERIs. For more information, see Instance families.
To determine the maximum number of ERIs that an instance type supports, check the value of the EriQuantity parameter in the response of the DescribeInstanceTypes operation. A value of 0 indicates that the instance type does not support ERIs.
When you attach multiple ERIs to an instance, consider the queue pair (QP) limit of the network interface controllers (NICs). For more information, see Queue pairs (QPs).
Create an Elastic RDMA Interface
Create an ERI with an instance
For instance types that support eRDMA, you can select Enable Elastic RDMA Interface when you purchase an instance. The specific configuration may vary based on the instance type:
Enterprise-level instances: For more information, see Configure eRDMA when you purchase an enterprise-level instance.
In the Elastic Network Interface section when creating an instance, select the Elastic RDMA Interface check box in the Primary ENI row. After you enable the Elastic RDMA Interface, IPv6 addresses are not supported.
GPU-accelerated instances: For more information, see Enable eRDMA on a GPU-accelerated instance.
In the Elastic Network Interface section, select the Elastic RDMA Interface check box for both the primary and secondary ENIs. To create a secondary ENI, click the Create Secondary ENI tab. Only one secondary ENI can be attached when creating an instance. Enabling eRDMA requires a specific OS.
Create an ERI separately
When you create a secondary ENI, enable eRDMA Interface. The Elastic RDMA Interface (ERI) shares the settings of the secondary ENI, such as its IP address and security group rules. For more information, see Create and use an ENI.
Create an ERI using OpenAPI
You can call the CreateNetworkInterface operation to create an ENI. To create an ENI with an ERI enabled, set the NetworkInterfaceTrafficMode parameter to HighPerformance.
For more information about how to manage ERIs, such as attaching, detaching, and modifying their properties, see Manage ENIs.
View an Elastic RDMA Interface
View in the console
You can check whether an ENI is RDMA-enabled in the console.
Go to ECS console - Instances.
In the upper-left corner of the page, select a region and resource group.
Find the target ECS instance and click the instance ID to go to the instance details page.
Click the ENIs tab to view the details of the ENIs attached to the ECS instance.
In the ENI Type column, the flexible RDMA interface flag indicates that an ERI is enabled for the NIC.
View using OpenAPI
You can call the DescribeNetworkInterfaceAttribute operation to query the properties of an ENI. The value of the NetworkInterfaceTrafficMode parameter in the response indicates whether an ERI is enabled for the ENI:
HighPerformance: The ERI is enabled and the elastic RDMA communication mode is used.
Standard: The ERI is disabled and the TCP communication mode is used.
View within an instance
After an ENI with an ERI is attached to an instance, you can view the RDMA network interface from within the instance.
Log on to the instance remotely.
For more information, see Log on to a Linux instance using Workbench.
Run the following command to view the RDMA network interface devices.
ibv_devicesThe device name in the output indicates that the device is an eRDMA network interface.
[root@xxx ~]# ibv_devices device node GUID ------ --------- erdma_0 02163effe16b4ffIf your instance has multiple ENIs with ERIs enabled, the output is similar to the following:
[root@xxx ~]# ibv_devices device node GUID ------ --------- erdma_0 02163efffe10008e erdma 1 02163efffe40d834
You can also run the
ibv_devinfocommand to view more configuration parameters of the ERI, such as the port status and the maximum number of supported queues.[root@xxx ~]# ibv_devinfo hca_id: erdma_0 transport: eRDMA (1) fw_ver: 0.2.0 node_guid: 0216:3eff:fe13:1914 sys_image_guid: 0216:3eff:fe13:1914 vendor_id: 0x1ded vendor_part_id: 4223 hw_ver: 0x0 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: EthernetIf the preceding information is not displayed, the eRDMA driver may be installed incorrectly or the network interface may be configured incorrectly. For more information about how to resolve the issue, see Verify the eRDMA configuration.
Modify the RDMA interface of an existing ENI
You can call the ModifyNetworkInterfaceAttribute operation and set the NetworkInterfaceTrafficMode parameter in NetworkInterfaceTrafficConfig to configure the communication mode of the ENI. This enables or disables the ERI.
HighPerformance: The ERI is enabled and the elastic RDMA communication mode is used.
Standard: The ERI is disabled and the TCP communication mode is used.
You cannot enable or disable the ERI for an existing ENI in the console.
Queue pairs (QPs)
A QP is a basic communication entity in RDMA. It consists of a Send Queue (SQ) and a Receive Queue (RQ). The QP is used to manage the data sent and received.
The QP number is the maximum number of QPs that can be created on an RDMA device or network interface. This number determines the maximum number of concurrent connections that the RDMA network can support and affects the network's extensibility and concurrent processing capabilities.
QP number for an instance
The number of QPs for an instance depends on its instance type. When you attach multiple ERIs to an ECS instance, the maximum total number of QPs for all ENIs varies based on the instance type. You can call the DescribeInstanceTypes operation to query the maximum number of QPs that an instance type allows for all its ENIs:
If the response contains NetworkCardQuantity, the maximum total number of QPs for all ENIs is QueuePairNumber * NetworkCardQuantity.
If the response does not contain NetworkCardQuantity, the maximum total number of QPs for all ENIs is QueuePairNumber.
Elastic RDMA NIC QP Count
If you attach an ERI to an instance without specifying the QueuePairNumber, the ERI uses the default value, which is the maximum total number of QPs allowed for the instance type. For instance types that support multiple ERIs, if you attach an ERI that uses this default value, you cannot attach more ERIs because the QP limit for the instance is reached. You must adjust the QP number for the ENI to free up capacity before you can attach another ERI to the instance.
Set the number of QPs for an ENI
You can call the ModifyNetworkInterfaceAttribute operation and set the QueuePairNumber parameter to the desired number of QPs for the ENI.
If the ENI is already attached to an instance, you must stop the instance before you can make adjustments.
For a secondary ENI, you must first detach it from the instance, modify the QueuePairNumber, and then attach the secondary ENI.
View the number of QPs for an ENI
You can call the DescribeNetworkInterfaceAttribute operation to query the properties of an ENI. The QueuePairNumber parameter in the response indicates the number of QPs for the ENI.
This parameter is not returned if the secondary ENI has been created but not yet attached to an instance.
Set the number of QPs appropriately
Setting an appropriate number of QPs is crucial for optimizing application performance. Too many or too few QPs can lead to resource waste or performance bottlenecks.
Understand application requirements: Determine the required number of QPs based on your application scenario, target instance type, and planned number of ENIs. If your application requires high concurrency and low latency, you may need more QPs to support more parallel connections. If your scenario involves large-scale data transmission, the bandwidth utilization of a single QP might be a more important factor.
Consider hardware limitations:
Different instance types allow different maximum numbers of QPs for ERIs. For more information about how to query this number, see QP number for an instance.
Consider factors such as the server memory size. Each QP consumes a certain amount of system resources.
Test and optimize: After the initial configuration, you should perform a performance test in your actual environment. You can then adjust the number of QPs and other related parameters based on the test results. Monitor for any abnormal conditions, such as error messages or an increased packet loss rate.
Continuously monitor: Even after you find a suitable configuration, you should regularly check the system's operational status. This ensures that the current settings remain appropriate as your business grows or your requirements change.
Setting the QP number correctly requires you to consider multiple factors. You must plan and adjust the settings carefully based on your business needs, existing infrastructure, and target performance.