Use eRDMA to accelerate container networks in ACK clusters

Enable eRDMA device access

Declare the aliyun/erdma resource in the container's resource limits:

spec:
  containers:
  - name: erdma-container
    resources:
      limits:
        aliyun/erdma: 1

After the pod starts, verify eRDMA devices are available:

/# ls /dev/infiniband/
rdma_cm  uverbs0

Enable transparent TCP acceleration with SMC-R

SMC-R transparently accelerates TCP connections over RDMA without code changes. After enabling eRDMA device access, add the following annotation:

metadata:
  annotations:
    network.alibabacloud.com/erdma-smcr: "true"

Accelerate NCCL communication for GPU distributed training

Use this scenario for GPU-accelerated instances running distributed training. Pods use the ofed driver and request GPU and eRDMA resources.

1. Install the ACK eRDMA Controller with preferDriver set to ofed.

2. Add GPU nodes to the node pool. See Create and manage node pools.

3. Install eRDMA packages in the container image at build time. For Debian or Ubuntu (replace {OS|ubuntu} and {Version|focal} with your values):

   wget -qO - https://mirrors.aliyun.com/erdma/GPGKEY | apt-key add - \
     && echo "deb [ arch=amd64 ] https://mirrors.aliyun.com/erdma/apt/{OS|ubuntu} {Version|focal}/erdma main" \
       | tee /etc/apt/sources.list.d/erdma.list \
     && apt update \
     && apt install -y libibverbs1 ibverbs-providers ibverbs-utils librdmacm1

   For Alibaba Cloud Linux or RHEL:

   cat > /etc/yum.repos.d/erdma.repo <<EOF
   [erdma]
   name = ERDMA Repository
   baseurl = http://mirrors.aliyun.com/erdma/yum/redhat/7/erdma/x86_64/
   gpgcheck = 0
   enabled = 1
   EOF
   yum install --disablerepo=* --enablerepo erdma -y libibverbs ibverbs-providers ibverbs-utils librdmacm

4. Deploy the GPU application. This StatefulSet runs nccl-test across two replicas, each with 8 GPUs and one eRDMA device:

   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: nccltest
   spec:
     selector:
       matchLabels:
         app: nccltest
     serviceName: "nccltest"
     replicas: 2
     template:
       metadata:
         labels:
           app: nccltest
       spec:
         hostNetwork: true
         dnsPolicy: ClusterFirstWithHostNet
         containers:
         - env:
           - name: NCCL_SOCKET_IFNAME
             value: "eth0"
           - name: NCCL_DEBUG
             value: "INFO"
           - name: NCCL_IB_GID_INDEX
             value: "1"
           image: <nccl-test-image-with-erdma>
           imagePullPolicy: Always
           name: nccltest
           securityContext:
             privileged: true
           resources:
             limits:
               nvidia.com/gpu: "8"
               aliyun/erdma: "1"
             requests:
               nvidia.com/gpu: "8"
               aliyun/erdma: "1"

5. Verify NCCL is using eRDMA. The logs should show erdma_0 and erdma_1 as active, confirming eRDMA acceleration.

   

Accelerate application networks transparently with SMC-R

Use this scenario for standard TCP workloads that need eRDMA acceleration without code changes. Set preferDriver to default.

1. Install the ACK eRDMA Controller with preferDriver set to default.

2. Deploy the application with the SMC-R annotation. This Deployment enables eRDMA device access and SMC-R acceleration:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: app-with-erdma
     name: app-with-erdma
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: app-with-erdma
     template:
       metadata:
         labels:
           app: app-with-erdma
         annotations:
           network.alibabacloud.com/erdma-smcr: "true"
       spec:
         containers:
         - image: <application image>
           imagePullPolicy: Always
           name: app-with-erdma
           resources:
             limits:
               aliyun/erdma: 1

3. Verify acceleration. Install smc-tools in the container and run smcss:

   /# smcss
   State          UID   Inode   Local Address           Peer Address            Intf Mode
   ACTIVE         00000 0059964 172.17.192.73:47772     172.17.192.10:80        0000 SMCR

   SMCR in the Mode column confirms eRDMA acceleration is active. If it shows TCP, verify both client and server pods have the SMC-R annotation set to "true".