Use RDMA on ACK Lingjun nodes-Container Service for Kubernetes(ACK)-阿里云帮助中心

Remote Direct Memory Access (RDMA) moves data directly between the memory of two computers without involving the operating system on either side. By bypassing the OS, RDMA eliminates the overhead from data copies and context switches — conserving memory bandwidth and CPU cycles. This makes RDMA well-suited for workloads that require high throughput and low latency, such as high-performance computing (HPC), AI training, and distributed storage.

Enable RDMA on ACK Lingjun pods

Step 1: Verify the RDMA Device Plugin is running

Run the following command to confirm the RDMA Device Plugin DaemonSet is running on your Lingjun nodes:

kubectl get ds ack-rdma-dp-ds -n kube-system

Expected output:

NAME             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
ack-rdma-dp-ds   2         2         2       2            2           <none>          xxh

When READY equals DESIRED, the RDMA Device Plugin is running on all target nodes.

Step 2: Verify the node exposes the RDMA resource

Check that the node advertises the rdma/hca extended resource:

kubectl get node e01-cn-xxxx -oyaml

Look for rdma/hca in the allocatable and capacity sections of the output:

  allocatable:
    cpu: 189280m
    ephemeral-storage: "3401372677838"
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 2063229768Ki
    nvidia.com/gpu: "8"
    pods: "64"
    rdma/hca: 1k
  capacity:
    cpu: "192"
    ephemeral-storage: 3690725568Ki
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 2112881480Ki
    nvidia.com/gpu: "8"
    pods: "64"
    rdma/hca: 1k

The rdma/hca: 1k entry in the allocatable section confirms the RDMA Device Plugin has registered the resource with the kubelet.

Step 3: Submit a Job that requests RDMA

Apply the following Job manifest. Two fields are required for RDMA to work:

rdma/hca: 1 — requests one RDMA device for the container.
hostNetwork: true — gives the pod direct access to the host network interface where the RDMA device is exposed. Without host network access, the pod cannot reach the RDMA device on the node.

apiVersion: batch/v1
kind: Job
metadata:
  name: hps-benchmark
spec:
  parallelism: 1
  template:
    spec:
      containers:
      - name: hps-benchmark
        image: <YOUR_IMAGE> # Replace with your actual registry address
        command:
        - sh
        - -c
        - |
          python /workspace/wdl_8gpu_outbrain.py
        resources:
          limits:
            nvidia.com/gpu: 8
            rdma/hca: 1
        workingDir: /root
        volumeMounts:
          - name: shm
            mountPath: /dev/shm
      restartPolicy: Never
      volumes:
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: 8Gi
      hostNetwork: true
      tolerations:
        - operator: Exists