What is the DeepNCCL AI communication acceleration library?

Product introduction

DeepNCCL is based on the NVIDIA Collective Communications Library (NCCL). It calls NCCL communication operators to provide more efficient communication for multi-GPU interconnects and seamlessly accelerate tasks such as distributed training or multi-GPU inference.

The following figure shows the architecture of DeepNCCL.

Optimization principles

The DeepNCCL communication acceleration library significantly optimizes communication for AI distributed training or multi-GPU inference tasks.

Single-machine optimization

Single-node optimization focuses on improving communication for instance types with different hardware topologies. The following sections describe optimizations for PCIe and NVLink interconnects.

• PCIe interconnect topology optimization: In this instance type, multiple GPU cards share PCIe bandwidth. Communication is often limited by the physical bandwidth. For PCIe interconnect topologies, the CPU-Reduce algorithm, which is a pipelined parameter server (PS) mode gradient reduction algorithm, can theoretically reduce communication time. This algorithm builds a pipeline from the GPU to the CPU and then back to the GPU. It distributes the gradient reduction computation across multiple devices to reduce communication bottlenecks.

  For example, in scenarios where the data volume exceeds 4 MB, the PCIe interconnect topology optimization provides a performance improvement of over 20% compared to native NCCL.

• NVLink interconnect topology optimization: The default Binary-Tree algorithm used by NCCL does not fully utilize the multi-channel performance on V100 instance types. For NVLink interconnect topologies, this optimization extends the combinations of different N-Trees topology structures within a single node. This allows for topology tuning and leverages multi-channel performance.

  For example, in scenarios where the data volume exceeds 128 MB, the NVLink interconnect topology optimization provides a performance improvement of over 20% compared to native NCCL.

Multi-node optimization

Multi-node optimization includes communication operator compilation optimization, TCP multi-stream optimization, and multi-node CPU-Reduce optimization.

• Communication operator compilation optimization: The Hybrid+ algorithm supports hierarchical communication for single and multiple nodes. This is different from algorithms such as Allreduce, Allgather, or Reduce-scatter, which are based on a global topology structure. The Hybrid+ algorithm is designed for the unique characteristics of different Alibaba Cloud instance types and the various topologies that connect network interface cards and GPUs. It fully utilizes the high internal bandwidth of a single node while reducing the amount of communication between multiple nodes. This optimization improves performance by over 50% compared to native NCCL.

• Communication multi-stream optimization: Network bandwidth is often underutilized. This prevents the cross-node performance of upper-layer collective communication algorithms from reaching its full potential. Using the TCP/IP-based multi-stream feature improves the concurrent communication capability of distributed training. This can increase multi-node training performance by 5% to 20%.

• Multi-node CPU-Reduce: This optimization inherits the efficient asynchronous pipeline of the single-node CPU-Reduce. It also formats the cross-node Socket communication as a pipeline. This creates a pipelined process for all multi-node communication, which reduces communication latency and improves overall training performance.

  For example, in multi-node training scenarios for Transformer-based models with large data volumes, the multi-node CPU-Reduce optimization can improve end-to-end performance by over 20%.

Performance

The DeepNCCL communication acceleration library improves performance for single-node Allreduce, multi-node Allreduce, multi-node Reduce-scatter, and multi-node Allgather operations.

References

You can install the DeepNCCL communication library on Elastic GPU Service instances to accelerate distributed training or inference performance. For more information, see Install and use DeepNCCL.