RDMA: distributed training with a high-performance network
Large-model AI parallel computing requires optimizations that reduce communication overhead, overlap computation with communication, and improve communication efficiency. RDMA (Remote Direct Memory Access) significantly reduces network latency and is well suited for distributed training jobs running on Lingjun intelligent computing resources.
Usage limits
This feature applies only to training jobs submitted on Lingjun intelligent computing resources.
Configure high-performance network variables
Platform for AI (PAI) enables the RDMA feature on Lingjun intelligent computing resources and preconfigures optimal NVIDIA Collective Communications Library (NCCL) variables. We recommend using the default variables for better performance. You can also adjust them based on your training framework and model characteristics.
Default variables (platform preset)
Depending on the Lingjun instance type, the platform preconfigures default variables. The following table lists the details:
|
Lingjun instance type |
NCCL variables |
|
|
For details about the NCCL variables, see Environment variable description.
Environment variable description
The following table describes key NCCL environment variables. For more information, see the NCCL documentation.
|
Key environment variable |
Description |
|
NCCL_IB_TC |
Matches the Alibaba Cloud network mapping rules. If this variable is not set or is set incorrectly, performance may degrade. |
|
NCCL_IB_GID_INDEX |
Provides a recommended value. If this variable is not set or is set incorrectly, NCCL may report errors. |
|
NCCL_SOCKET_IFNAME |
Selects the correct port for establishing connections. Different instance types have different port requirements. If this variable is not set or is set incorrectly, NCCL may fail to establish connections. |
|
NCCL_DEBUG |
Usually set to INFO to produce more NCCL-related log output, which helps identify and resolve potential issues. |
|
NCCL_IB_HCA |
Specifies the network interface controller (NIC) for RDMA communication. The number and naming convention of InfiniBand devices vary across compute nodes. If this variable is not set or is set incorrectly, performance may degrade. |
|
NCCL_IB_TIMEOUT |
Increases the RDMA connection timeout to improve fault tolerance for training jobs. If this variable is not set or is set incorrectly, training jobs may be interrupted. |
|
NCCL_IB_QPS_PER_CONNECTION |
Increases the number of Queue Pairs (QPs) per connection to improve network throughput. |
Configure images
When you submit a training job based on Lingjun intelligent computing resources, use the official built-in images provided by DLC.
The following three official GPU training images are available:
deepspeed-training:23.06-gpu-py310-cu121-ubuntu22.04
megatron-training:23.06-gpu-py310-cu121-ubuntu22.04
nemo-training:23.06-gpu-py310-cu121-ubuntu22.04
All three images include PyTorch 2.1, Megatron-LM 23.06, DeepSpeed 0.9.5, Transformers 4.29.2, NVIDIA NeMo 1.19.0, CUDA 12.1, Ubuntu 22.04, and Python 3.10. The images are deployed in the China (Ulanqab) region.
Related documents
For the detailed steps to submit a training job on Lingjun intelligent computing resources, see Create a training job.