Run applications using high-performance RDMA networks-Container Service for Kubernetes(ACK)-阿里云帮助中心

In large-scale AI training workloads, GPU communication overhead often limits overall performance. By combining ACK One registered clusters with Container Compute Service (ACS), you get a Remote Direct Memory Access (RDMA) network that delivers the low latency and high throughput your distributed training tasks need — without scaling out your cluster.

How it works

Most applications use TCP/IP, which introduces overhead through its complex protocol stack, high data copy costs, and frequent context switches. These factors make TCP/IP a bottleneck for GPU-intensive workloads.

RDMA bypasses these bottlenecks with two key mechanisms:

Zero-copy transfers: Data moves directly between application memory spaces, eliminating data copy overhead.
Kernel bypass: Network operations skip the operating system kernel, removing context switches from the data path.

The result is lower latency, reduced CPU usage, and higher throughput for inter-GPU communication.

To run a pod on an RDMA network, add the following label to your pod spec:

alibabacloud.com/hpn-type: "rdma"

GPU models that support RDMA

ACS offers multiple GPU options. For High-Performance Network (HPN) RDMA capabilities, deploy the 8th-gen GPU A GPU. To verify compatibility with other GPU models, submit a ticket to contact support.

Prerequisites

Before you begin, ensure that you have:

An ACK One registered cluster connected to a data center or a Kubernetes cluster from another cloud service provider (Kubernetes 1.24 or later recommended). See Create an ACK One registered cluster.
The ack-virtual-node component at version 2.13.0 or later. To set up RAM permissions and install the component, see Grant RAM permissions to ack-virtual-node and Install ack-virtual-node.

Deploy an application on an RDMA network

Create a file named dep-demo-hpn-gpu.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dep-demo-hpn-gpu
  labels:
    app: demo-hpn-gpu
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-hpn-gpu
  template:
    metadata:
      labels:
        app: demo-hpn-gpu
        alibabacloud.com/acs: "true"              # Use ACS compute resources
        alibabacloud.com/compute-class: gpu
        alibabacloud.com/compute-qos: default
        # Replace example-model with your actual GPU model series
        alibabacloud.com/gpu-model-series: "example-model"
        alibabacloud.com/hpn-type: "rdma"         # Enable RDMA network
    spec:
      containers:
      - name: demo
        image: registry.cn-wulanchabu.aliyuncs.com/acs/stress:v1.0.4
        command:
        - "sleep"
        - "1000h"
        resources:
          requests:
            cpu: 128
            memory: 512Gi
            nvidia.com/gpu: 8
          limits:
            cpu: 128
            memory: 512Gi
            nvidia.com/gpu: 8

Deploy the application:
```
kubectl apply -f dep-demo-hpn-gpu.yaml
```

Verify that the pod has an RDMA network interface card (NIC):

kubectl exec -it dep-demo-hpn-gpu-xxxxx-xxx -- ifconfig | grep hpn -A 8

The expected output is similar to:

hpn0      Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:xx
          inet6 addr: xxxx::x:xxxx:xxxx:xxx/xx Scope:Link
          inet6 addr: xxxx:xxx:xxx:x:x:xxxx:x:xxx/xxx Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:xxxx  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:xx errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:x (892.0 B)

The hpn0 interface confirms that the pod is connected to the RDMA network. The UP BROADCAST RUNNING flags indicate the interface is active and ready to handle traffic.