After you attach an Elastic RDMA Interface (ERI) to a GPU instance, GPU instances in the same VPC use RDMA passthrough to accelerate interconnectivity. eRDMA improves on traditional RDMA by delivering more efficient data transfer, improving communication efficiency between GPU instances, and reducing task processing time. This topic explains how to enable eRDMA on a GPU instance.
Limitations
Item | Description |
Instance type | The elastic RDMA interface is supported on the following instance types:
|
Image | Use one of the following images:
|
Number of elastic RDMA interfaces |
|
Network limitations |
|
Procedure
To use eRDMA on an eRDMA-capable instance, two requirements must be met: the eRDMA software stack must be installed, and an elastic network interface (ENI) with the Elastic RDMA Interface (ERI) enabled must be attached.
Configure eRDMA at instance creation
Go to the Custom Launch page in the ECS console.
Create a GPU instance that supports ERI.
When you create the instance, note the following settings. For information about other parameters, see Create an instance by using the wizard.
Instance Type: Select an ERI-capable instance type. For more information, see Limits. This topic uses ebmgn7ex or ebmgn8is as an example.
Images: When you select a public image, the Auto-install GPU Driver and Install eRDMA Software Stack options are selected by default. After the instance is created, the system automatically installs the GPU driver, CUDA, cuDNN, and the eRDMA software stack.
In this example, the operating system is Alibaba Cloud Linux 3.2104 LTS 64-bit, and the GPU driver version is CUDA Version 12.4.1 / Driver Version 550.127.08 / CUDNN Version 9.2.0.82.
(Optional) Jumbo Frames: If the selected instance supports jumbo frames, you can enable this feature to improve
eRDMAcommunication performance.Enabling jumbo frames allows you to set a larger maximum transmission unit (MTU). When you use NCCL and the LL128 low-latency protocol for communication, the MTU must be 8500. If jumbo frames are not enabled, the MTU should be 1400. An incorrect MTU setting can cause data consistency issues.
ENIs: When you create a GPU instance and configure ENIs on the Bandwidths & Security Groups page of the configuration wizard, an eRDMA-capable primary NIC and secondary NIC are created by default. The system automatically selects the eRDMA Interface option for the primary and secondary NICs.
NoteYou cannot enable or disable the ERI capability for a single ENI while the GPU instance is running.
The two ERI-enabled NICs are automatically bound to different channels. You do not need to manually specify the channels.
The primary NIC cannot be detached from the GPU instance. It is created and released with the instance.
Go to the created instance's details page and click the ENIs tab to view the instance's NIC type.
If the NIC type of the primary NIC or a secondary ENI includes "Elastic RDMA Interface", the ENI has ERI enabled.
Configure eRDMA for an existing GPU instance
Log on to the ECS console.
Find the target instance, go to its details page, and click the ENIs tab to check whether ERI is enabled.
If an ENI is ERI-enabled, the ENI Type column displays Primary ENI (Elastic RDMA Interface) or Secondary ENI (Elastic RDMA Interface).
If ERI is enabled, skip the following steps.
If ERI is not enabled, follow these steps to configure eRDMA for the primary NIC or a secondary ENI.
Configure eRDMA for the primary NIC or a secondary ENI.
NoteFor instance types that support jumbo frames, you can enable jumbo frames to improve
eRDMAcommunication performance.Enabling jumbo frames allows you to set a larger MTU. When you use NCCL and the LL128 low-latency protocol for communication, the MTU must be 8500. If jumbo frames are not enabled, the MTU should be 1400. An incorrect MTU setting can cause data consistency issues.
If you did not select the eRDMA Interface option for either the primary NIC or the secondary NIC when you created the GPU instance, you can create and enable two ERI-enabled secondary ENIs after the instance is created.
If you did not select the eRDMA Interface option for one of the two NICs (primary or secondary) when you created the GPU instance, you can create and enable only one more ERI-enabled secondary ENI after the instance is created.
(Optional) Install the eRDMA software stack in the instance.
If you did not select the Install eRDMA Software Stack option when you selected the public image, you must install the eRDMA software stack manually or with a script to enable the Elastic RDMA Interface (ERI) capability.
Script installation
After the GPU instance is created, you can use the following example script to separately install the eRDMA software stack, GPU driver, CUDA, and cuDNN.
#!/bin/sh #Please input version to install DRIVER_VERSION="570.133.20" CUDA_VERSION="12.8.1" CUDNN_VERSION="9.8.0.87" IS_INSTALL_eRDMA="TRUE" IS_INSTALL_RDMA="FALSE" INSTALL_DIR="/root/auto_install" #using .run to install driver and cuda auto_install_script="auto_install_v4.0.sh" script_download_url=$(curl http://100.100.100.200/latest/meta-data/source-address | head -1)"/opsx/ecs/linux/binary/script/${auto_install_script}" echo $script_download_url rm -rf $INSTALL_DIR mkdir -p $INSTALL_DIR cd $INSTALL_DIR && wget -t 10 --timeout=10 $script_download_url && bash ${INSTALL_DIR}/${auto_install_script} $DRIVER_VERSION $CUDA_VERSION $CUDNN_VERSION $IS_INSTALL_RDMA $IS_INSTALL_eRDMAManual installation
After you create a GPU instance, you can manually install the OFED driver, eRDMA driver, and GPU driver, and then load the
nv_peer_memservice component. Follow these steps:Connect to the GPU instance.
For more information, see Connect to a Linux instance by using Workbench.
Install the OFED driver.
Run the following command to install the required packages.
Alibaba Cloud Linux 3
yum install rpm-build flex iptables-devel systemd-devel gdb-headless elfutils-devel python3-Cython bison numactl-devel libmnl-devel libnl3-devel libdb-devel libselinux-devel perl-generators elfutils-libelf-devel kernel-rpm-macros valgrind-devel cmake lsof -yCentOS 8.5/8.4/7.9
CentOS 8.5/8.4
wget http://mirrors.cloud.aliyuncs.com/opsx/ecs/linux/binary/erdma/centos8/python3-Cython-0.29.32-3.16.x86_64.rpm yum install python3-Cython-0.29.32-3.16.x86_64.rpm -y yum install kernel-rpm-macros perl-generators libmnl-devel valgrind-devel rpm-build systemd-devel libdb-devel iptables-devel lsof elfutils-devel bison libnl3-devel libselinux-devel flex cmake numactl-devel -yCentOS 7.9
sudo yum install python-devel python3-Cython kernel-rpm-macros perl-generators libmnl-devel valgrind-devel rpm-build systemd-devel libdb-devel iptables-devel lsof elfutils-devel bison libnl3-devel libselinux-devel flex cmake numactl-devel -y
Ubuntu 24.04/22.04/20.04/18.04
Ubuntu 24.04
sudo apt-get update -y sudo apt-get install -y pkg-configUbuntu 22.04
sudo apt-get update -y sudo apt-get install -y pkg-configUbuntu 20.04
sudo apt-get update -y sudo apt-get install -y pkg-configUbuntu 18.04
sudo apt-get update sudo apt-get install -y pkg-config sudo apt install -y make dh-python libdb-dev libselinux1-dev flex dpatch swig graphviz chrpath quilt python3-distutils bison libmnl-dev libelf-dev gcc sudo python3
Run the following command to download the OFED package configuration file.
Alibaba Cloud Linux 3
sudo wget http://mirrors.cloud.aliyuncs.com/erdma/kernel-fix/MLNX_OFED_SRC-24.10-3.2.5.0.tgz sudo tar -xvf MLNX_OFED_SRC-24.10-3.2.5.0.tgz && cd MLNX_OFED_SRC-24.10-3.2.5.0 sudo wget http://mirrors.cloud.aliyuncs.com/opsx/ecs/linux/binary/erdma/ofed/alibaba_cloud3/3/ofed_alibaba_cloud3.conf sudo rm -rf SRPMS/mlnx-ofa_kernel-24.10-OFED.24.10.3.2.5.1.src.rpm sudo wget http://mirrors.cloud.aliyuncs.com/erdma/kernel-fix/mlnx-ofa_kernel-24.10-OFED.24.10.3.2.5.1.egs.1.src.rpm -O SRPMS/mlnx-ofa_kernel-24.10-OFED.24.10.3.2.5.1.egs.1.src.rpmCentOS 8.5/8.4/7.9
CentOS 8.5/8.4
cd /root wget http://mirrors.cloud.aliyuncs.com/opsx/ecs/linux/binary/erdma/ofed/MLNX_OFED_SRC-5.4-3.5.8.0.tgz tar -xvf MLNX_OFED_SRC-5.4-3.5.8.0.tgz && cd MLNX_OFED_SRC-5.4-3.5.8.0/ wget http://mirrors.cloud.aliyuncs.com/opsx/ecs/linux/binary/erdma/ofed/alibaba_cloud3/3/ofed_alibaba_cloud3.conf rm -rf SRPMS/mlnx-ofa_kernel-5.4-OFED.5.4.3.5.8.1.src.rpm wget http://mirrors.cloud.aliyuncs.com/erdma/kernel-fix/mlnx-ofa_kernel-5.4-OFED.5.4.3.5.8.1.egs.1.src.rpm -O SRPMS/mlnx-ofa_kernel-5.4-OFED.5.4.3.5.8.1.egs.1.src.rpmCentOS 7.9
sudo wget http://mirrors.cloud.aliyuncs.com/opsx/ecs/linux/binary/erdma/ofed/MLNX_OFED_SRC-5.4-3.5.8.0.tgz sudo tar -xvf MLNX_OFED_SRC-5.4-3.5.8.0.tgz && cd MLNX_OFED_SRC-5.4-3.5.8.0/ sudo wget http://mirrors.cloud.aliyuncs.com/opsx/ecs/linux/binary/erdma/ofed/alibaba_cloud3/3/ofed_alibaba_cloud3.conf sudo rm -rf SRPMS/mlnx-ofa_kernel-5.4-OFED.5.4.3.5.8.1.src.rpm sudo wget http://mirrors.cloud.aliyuncs.com/erdma/kernel-fix/mlnx-ofa_kernel-5.4-OFED.5.4.3.5.8.1.egs.1.src.rpm -O SRPMS/mlnx-ofa_kernel-5.4-OFED.5.4.3.5.8.1.egs.1.src.rpm
Ubuntu 24.04/22.04/20.04/18.04
Ubuntu 24.04/22.04/20.04
sudo wget http://mirrors.cloud.aliyuncs.com/erdma/kernel-fix/deb/MLNX_OFED_SRC-debian-24.10-3.2.5.0.tgz sudo tar -xvf MLNX_OFED_SRC-debian-24.10-3.2.5.0.tgz && cd MLNX_OFED_SRC-24.10-3.2.5.0 && curl -O http://mirrors.cloud.aliyuncs.com/erdma/kernel-fix/deb/ofed_debian.conf sudo rm -rf SOURCES/mlnx-ofed-kernel_24.10.OFED.24.10.3.2.5.1.orig.tar.gz wget http://mirrors.cloud.aliyuncs.com/erdma/kernel-fix/deb/mlnx-ofed-kernel_24.10.egs.1.OFED.24.10.3.2.5.1.orig.tar.gz -O SOURCES/mlnx-ofed-kernel_24.10.egs.1.OFED.24.10.3.2.5.1.orig.tar.gzUbuntu 18.04
sudo wget http://mirrors.cloud.aliyuncs.com/opsx/ecs/linux/binary/erdma/ofed/MLNX_OFED_SRC-debian-5.4-3.6.8.1.tgz sudo tar -xvf MLNX_OFED_SRC-debian-5.4-3.6.8.1.tgz && cd MLNX_OFED_SRC-5.4-3.6.8.1 && curl -O http://mirrors.cloud.aliyuncs.com/erdma/kernel-fix/deb/ofed_debian.conf sudo rm -rf SOURCES/mlnx-ofed-kernel_5.4.orig.tar.gz sudo wget http://mirrors.cloud.aliyuncs.com/erdma/kernel-fix/deb/mlnx-ofed-kernel_5.4.egs.orig.tar.gz -O SOURCES/mlnx-ofed-kernel_5.4.egs.orig.tar.gz
Run the command for your OS to install the OFED driver.
Alibaba Cloud Linux 3
sudo ./install.pl --config ./ofed_alibaba_cloud3.conf --distro RHEL8 sudo dracut -fCentOS 8.5/8.4/7.9
CentOS 8.5/8.4
./install.pl --config ./ofed_alibaba_cloud3.conf --distro RHEL8CentOS 7.9
sudo ./install.pl --config ./ofed_alibaba_cloud3.conf --distro RHEL7
Ubuntu 24.04/22.04/20.04/18.04
Replace
${VERSION_ID}with your Ubuntu version, such as 24.04.sudo curl -O http://mirrors.cloud.aliyuncs.com/erdma/kernel-fix/deb/ofed_debian.conf sudo ./install.pl --config ./ofed_debian.conf --without-dkms --build-only --kernel-only sudo /usr/bin/dpkg -i --force-confmiss DEBS/ubuntu`lsb_release -s -r`/x86_64/*.deb update-initramfs -uRun the following command to check whether the
/usr/src/ofa_kernel/`uname -r`directory exists.If the directory exists, proceed to the next step.
ls /usr/src/ofa_kernel/`uname -r`If the directory does not exist, run the following command to create a symbolic link, and then proceed to the next step.
sudo ln -s /usr/src/ofa_kernel/default /usr/src/ofa_kernel/`uname -r`
Restart the instance.
After installing the OFED driver, restart the instance for the new kernel module to take effect. For more information, see Restart an instance.
Install the eRDMA driver.
Download and install the eRDMA driver.
sudo wget http://mirrors.cloud.aliyuncs.com/erdma/env_setup.sh sudo bash env_setup.sh --egsRun the following command to verify the eRDMA driver installation by using the eadm tool:
eadm verAn output similar to the following indicates a successful installation.
[root@xxx ~]# eadm ver Query kernel driver version: 0.2.35NoteThis topic uses driver version 0.2.35 as an example. If the command returns a "command not found" error or fails to run, reinstall the eRDMA driver.
Install the GPU driver.
For more information, see Manually install a Tesla driver (Linux) on a GPU instance.
Load the nv_peer_mem service component.
(Recommended) GPU driver 470.xx.xx or later
To enable GPUDirect RDMA, you must load the nv_peer_mem service component. We recommend that you use GPU driver 470.xx.xx or later because NVIDIA includes this component in these driver versions. You can run the following command to load the nvidia_peermem module.
sudo modprobe nvidia_peermem # You can run lsmod|grep nvidia to check whether nvidia_peermem is loaded.NoteIf the instance is restarted, you must reload the nvidia_peermem module.
GPU drivers earlier than 470.xx.xx
You must manually download and install the service component. The following code shows how to download, compile, and install the component.
sudo git clone https://github.com/Mellanox/nv_peer_memory.git # Compile and install nv_peer_mem.ko. cd nv_peer_memory && make cp nv_peer_mem.ko /lib/modules/$(uname -r)/kernel/drivers/video depmod -a modprobe nv_peer_mem # You can run lsmod|grep nv_peer_mem to check the result. service nv_peer_mem start
Verify the bandwidth.
Connect to the GPU instance.
For more information, see Connect to a Linux instance by using Workbench.
Run the following command to check whether the two eRDMA NICs are working as expected.
sudo ibv_devinfoBy default, the eRDMA driver installation script installs the latest driver version. If you need to install an earlier version of the eRDMA driver, submit a ticket for assistance.
This topic uses eRDMA driver version 0.2.37 or later as an example. The following output shows two eRDMA devices working correctly. An eRDMA device is working correctly if its port
stateisPORT_ACTIVE.[ecs-xxx...xxx4gnd0hZ ~]$ sudo ibv_devinfo hca_id: erdma_0 transport: eRDMA (0) fw_ver: 0.2.0 node_guid: 0216:3eff:fe36:1eb4 sys_image_guid: 0216:3eff:fe36:1eb4 vendor_id: 0x1ded vendor_part_id: 4223 hw_ver: 0x0 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 1024 (3) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: erdma_1 transport: eRDMA (0) fw_ver: 0.2.0 node_guid: 0216:3eff:fe43:9c2a sys_image_guid: 0216:3eff:fe43:9c2a vendor_id: 0x1ded vendor_part_id: 4223 hw_ver: 0x0 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 1024 (3) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: EthernetNoteIf the port
stateisinvalid state, the device is not working correctly. We recommend that you first check the secondary ENI configuration. For example, run theifconfigcommand to verify that each NIC has the correct configuration and IP address.Run the following command to install the perftest tool.
sudo yum install perftest -yRun the following commands to test whether the RDMA network bandwidth meets hardware expectations.
On the server, run the following command to listen for client connection requests.
sudo ib_write_bw -d erdma_0 -F -q 16 --run_infinitely --report_gbits -p 18515On the client, run the following command to send connection requests and data packets.
sudo ib_write_bw -d erdma_0 -F -q 16 --run_infinitely --report_gbits -p 18515 server_ipIn this command,
server_ipis the private IP address of the server's ERI-enabled ENI. To find this IP address, see View IP addresses.
NoteThe preceding perftest benchmark uses one NIC for communication. If your workload requires two NICs for communication, you must start two perftest processes. Then, use the
-dparameter to specify an eRDMA NIC for each process and the-pparameter to specify different communication ports. For more information, see perftest Details.The test results include the average bandwidth. An output similar to the following indicates that eRDMA communication is normal.
Test and verification
This topic uses nccl-tests as an example to demonstrate how to test and verify the performance of eRDMA-enabled GPU instances. For more information about nccl-tests, see nccl-tests.
Run the following commands to install NCCL.
Compile and install NCCL from the source code:
NoteYou can also download and install the package from the official NVIDIA NCCL website.
This example installs NCCL to the
/usr/local/nccldirectory. You can specify a different installation path based on your requirements.# build nccl cd /root git clone https://github.com/NVIDIA/nccl.git cd nccl/ make -j src.lib PREFIX=/usr/local/nccl make install PREFIX=/usr/local/ncclRun the following commands to verify that NCCL and the libnccl.so library are installed.
# Check for NCCL ls /usr/local/nccl # Check for the libnccl.so library ls /usr/local/nccl/libRun the following commands to install Open MPI and the required compilers.
wget https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.3.tar.gz tar -xzf openmpi-4.1.3.tar.gz cd openmpi-4.1.3 ./configure --prefix=/usr/local/openmpi make -j && make installRun the following commands to set environment variables.
NCCL_HOME=/usr/local/nccl CUDA_HOME=/usr/local/cuda MPI_HOME=/usr/local/openmpi export LD_LIBRARY_PATH=${NCCL_HOME}/lib:${CUDA_HOME}/lib64:${MPI_HOME}/lib:$LD_LIBRARY_PATH export PATH=${CUDA_HOME}/bin:${MPI_HOME}/bin:$PATHThe preceding commands use the following example paths:
NCCL_HOMEpoints to the NCCL installation path (/usr/local/nccl),CUDA_HOMEpoints to the CUDA installation path (/usr/local/cuda), andMPI_HOMEpoints to the Open MPI installation path (/usr/local/openmpi). Replace these paths with your actual installation paths.After you finish editing the command, edit the
~/.bashrcfile on the instance to setPATHandLD_LIBRARY_PATH. Then, run the following command for the environment variable settings to take effect.source ~/.bashrcRun the following commands to download and compile the test code.
git clone https://github.com/NVIDIA/nccl-tests cd nccl-tests/ make MPI=1 CUDA_HOME=/usr/local/cuda MPI_HOME=/usr/local/openmpiRun the following commands to set up passwordless SSH access between instances.
To set up passwordless SSH access, generate a public key on host1 and copy it to host2.
# On host1 ssh-keygen ssh-copy-id -i ~/.ssh/id_rsa.pub ${host2} ssh root@${host2} # On host1, run this command to test the connection. A successful login without a password prompt confirms that the setup is complete.Run the following command to test the NCCL all-reduce performance.
# Replace host1 and host2 with the IP addresses of your instances. mpirun --allow-run-as-root -np 16 -npernode 8 -H host1:8,host2:8 \ --bind-to none \ -mca btl_tcp_if_include eth0 \ -x NCCL_SOCKET_IFNAME=eth0 \ -x NCCL_GIN_TYPE=0 \ -x NCCL_DEBUG=INFO \ -x LD_LIBRARY_PATH \ -x PATH \ ./build/all_reduce_perf -b 4M -e 4M -f 2 -g 1 -t 1 -n 20
eRDMA configuration verification
After you complete the basic eRDMA configuration, use the following checklist to verify your environment and ensure that eRDMA functions correctly.
This section applies to the following eRDMA-capable GPU elastic bare metal instance types: ecs.ebmgn9g, ecs.ebmgn9gc, ecs.ebmgn9ge, ecs.ebmgn8is, and ecs.ebmgn8v.
Category | Item | Verification method | Description |
Network interface configuration | The instance has two eRDMA network interfaces | Run the | When you create an instance from the console, two eRDMA network interfaces are configured by default. No further action is required. |
Network interface state is PORT_ACTIVE | Run the following command to verify that the status of all ports is | If the check fails, run the | |
The two network interfaces are attached to different NUMA nodes | Run the following command and verify that the return values are | If both commands return 0, the secondary network interface is attached to an incorrect NUMA node. To fix this, follow these steps:
| |
Jumbo frames are enabled | Run the following command and verify that the MTU of the network interfaces is 4096: | If the MTU is not 4096, enable jumbo frames and then run the check again. | |
The MPCC congestion control algorithm is enabled | Run the following commands to check the congestion control algorithm: | eRDMA driver 1.5.6 and later use the MPCC algorithm by default. If MPCC is not enabled, run the following commands to enable it manually: | |
No IP address conflicts exist for the eRDMA network interfaces | Run the | This issue often occurs when you use the ACK Terway network plug-in. If you encounter this issue, see Terway whitelist configuration for solutions. | |
NCCL | Specify the NCCL topology file | Save the topology file (l20n.xml) to a local path on the instance, such as Before you start an NCCL task, set the following environment variable: | Topology file download URLs: |
GPU | Disable GPU ACS (PCI Access Control Services) to improve P2P communication performance | Run the following command to check the ACS status: If | If |
References
Configure eRDMA on an enterprise-level instance for an ultra-low latency, high-throughput, and elastic RDMA network service without changing your network topology. For more information, see Enable eRDMA on enterprise-level instances.
For applications that require large-scale data transfers and high-performance network communication in containers, you can integrate eRDMA into a container (Docker) environment. This process lets containerized applications bypass the OS kernel and directly access the host's eRDMA devices, enabling faster data transfer and more efficient communication. For more information, see Enable eRDMA in a container (Docker).
To monitor and diagnose eRDMA, see Monitor and diagnose eRDMA.