training-nv-pytorch 25.11

Main features

• This release adds multi-architecture support for amd64 and aarch64, supports CUDA 13.0.2 and CUDA 12.8.

• This release upgrades core components, including PyTorch, to version 2.9.

• This release upgrades core training components: Transformers to 4.57.1, DeepSpeed to 0.18.1, and TransformerEngine to 2.8 with support for Qwen3-VL.

• This release upgrades the core inference component vLLM to 0.11.2.

Bug fixes

No bug fixes are included in this release.

Assets

Driver requirements

• Release 25.11 supports CUDA 12.8.0 and CUDA 13.0.2, depending on your driver version. CUDA 13.0.2 requires NVIDIA driver version 580 or later, and CUDA 12.8.0 requires NVIDIA driver version 575 or later. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

PyTorch compilation optimization

torch.compile(), introduced in PyTorch 2.0, often delivers strong gains for small-scale, single-GPU workloads. But LLM training depends on GPU memory optimization and distributed frameworks such as FSDP or DeepSpeed, so torch.compile() may offer limited benefits or even degrade performance.

• Control communication granularity in the DeepSpeed framework. This helps the compiler capture a more complete compute graph and apply broader compilation optimizations.

• Use an optimized PyTorch build:

  • The PyTorch compiler frontend is improved to ensure that compilation succeeds even if a graph break occurs in the compute graph.

  • Pattern matching and dynamic shape support are strengthened to improve post-compilation performance.

With these optimizations, 8B-parameter LLM training typically achieves an end-to-end throughput gain of about 20%.

GPU memory optimization for recomputation

A predictive model for GPU memory overhead, built on large-scale performance data—including different models, clusters, and training parameter settings, as well as system metrics such as GPU memory utilization collected during benchmarking—recommends the optimal number of activation recomputation layers. This approach is integrated into PyTorch, allowing you to achieve the performance gains of GPU memory optimization with minimal effort. This feature is now supported in the DeepSpeed framework.

E2E performance evaluation

We used CNP (cloud-native AI performance benchmarking tool) to perform a comprehensive end-to-end performance comparison between our image and a standard base image using mainstream open-source models and framework configurations. We also performed an ablation study to evaluate the performance contribution of each optimized component to the overall model training.

Image performance evaluation: baseline comparison and iterative analysis

E2E performance contribution of core GPU components

The following tests are based on the 25.11 release and were conducted on a multi-node GPU cluster to evaluate and compare end-to-end training performance. The comparison includes:

1. Base: The NGC PyTorch image.

2. ACS AI Image: Base+ACCL: The image uses the ACCL communication library.

3. ACS AI Image: AC2+ACCL: This image uses the AC2 BaseOS with no optimizations enabled.

4. ACS AI Image: AC2+ACCL+CompilerOpt: This image uses the AC2 BaseOS with only torch.compile() optimization enabled.

5. ACS AI Image: AC2+ACCL+CompilerOpt+CkptOpt: This image uses the AC2 BaseOS with both torch.compile() and selective gradient checkpointing (CkptOpt) enabled.

1. Select an image

docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/training-nv-pytorch:[tag]

2. Enable compiler optimization and GPU memory optimization

• Enable compiler optimization

  Use the Transformers Trainer API:

  training_args = TrainingArguments(
      bf16=True,
      gradient_checkpointing=True,
      torch_compile=True
  )

• Enable GPU memory optimization for recomputation

  export CHECKPOINT_OPTIMIZATION=true

3. Start the container

The image includes a built-in model training tool, ljperf. The following steps show how to start the container and use this tool to run a training task.

# Start the container and enter its shell.
docker run --rm -it --ipc=host --net=host  --privileged egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/training-nv-pytorch:[tag]
# Run the training demo.
ljperf benchmark --model deepspeed/llama3-8b 

4. Usage notes

• This image includes changes to libraries such as PyTorch and DeepSpeed. Do not reinstall them.

• In your DeepSpeed configuration, leave zero_optimization.stage3_prefetch_bucket_size empty or set it to auto.

• Set the built-in environment variable NCCL_SOCKET_IFNAME based on your use case:

  • If a single pod requests 1, 2, 4, or 8 GPUs for a training or inference task, set NCCL_SOCKET_IFNAME=eth0. This is the default setting in the image.

  • If a single pod requests all 16 GPUs on a node for a training or inference task (which allows you to use the High-Performance Network (HPN)), set NCCL_SOCKET_IFNAME=hpn0.