training-xpu-pytorch 26.01

Main features

• Upgraded the PPU SDK to version 2.0.0.

• Upgraded transformer_engine to 2.8 and the vLLM inference component to 0.12.0+ppu2.0.0.oe.

• Updated health_check to be compatible with shuttle 1.5.3.

• Added a new image compatible with vLLM 0.14.0, which includes upgrades to torch 2.9, triton 3.5, deepspeed 0.18.5, and transformers 4.57.6.

• Upgraded vLLM to 0.16.0, which also upgrades Flashinfer to 0.6.3 and megatron-core to 0.16.0.

Bug fixes

• Fixed the following issues in vLLM 0.16.0:

  • Fixed garbled output in GLM-5 responses.

  • Fixed a server failure when running the Qwen3.5-35B-A3B model with TP=2.

Assets

Driver requirements

• Driver version >= 1.1.0

PyTorch compilation optimization

The compilation optimization feature introduced in PyTorch 2.0 usually provides significant benefits in small-scale, single-GPU scenarios, but in LLM training, the need to incorporate memory optimization and distributed frameworks such as FSDP and Deepspeed makes it difficult for torch.compile() to provide benefits and can even cause performance degradation:

• This optimization controls communication granularity within the DeepSpeed framework, allowing the compiler to capture a more complete computation graph and apply broader optimizations.

• This release includes an optimized version of PyTorch with the following enhancements:

  • An optimized PyTorch compiler frontend that ensures successful compilation even when arbitrary graph breaks occur.

  • Enhanced pattern matching and dynamic shape capabilities to improve the performance of compiled code.

With these optimizations, you can typically achieve an end-to-end (E2E) throughput gain of around 20% when training an 8B LLM.

Memory optimization with gradient checkpointing

A model to predict memory overhead was built using extensive benchmark data from various models, clusters, and training configurations—along with system metrics like memory utilization. This allows the system to recommend the optimal number of activation checkpointing layers. Integrated into PyTorch and compatible with the DeepSpeed framework, this feature helps users optimize memory usage.

ACCL communication library

ACCL is a high-performance network communication library developed by Alibaba for Lingjun products. It is available in three versions: ACCL-N for NVIDIA GPUs, ACCL-P for PPUs, and ACCL-R for AMD accelerators. ACCL-N is based on NVIDIA NCCL, offering full compatibility and including bug fixes and optimizations for performance and stability. ACCL-P is a collective communication library built upon T-Head's open-source pccl. ACCL-R is built upon AMD's open-source ROCm rccl. This release ports key features from ACCL and ACCL-N to pccl, resolves known issues, and is deeply customized for integration with Alibaba Cloud's proprietary networking components.

End-to-end performance evaluation

Using CNP, a cloud-native AI performance evaluation and analysis tool, we conducted a comprehensive end-to-end performance comparison against the standard base image with mainstream open-source models and framework configurations. We also performed an ablation study to evaluate the performance contribution of each optimization component to the overall model training performance.

Image comparison with base image and iterative evaluation

E2E performance contribution analysis of PPU and GPU core components

The following tests were based on version 26.01 and conducted on a multi-node PPU cluster to evaluate and compare end-to-end training performance. The comparison items include:

1. Base: The PPU PyTorch base image.

2. ACS AI Image: Base+ACCL: The base image using the ACCL communication library.

3. ACS AI Image: AC2+ACCL: The golden image using AC2 BaseOS with no optimizations enabled.

4. ACS AI Image: AC2+ACCL+CompilerOpt: The golden image using AC2 BaseOS with only the torch.compile optimization enabled.

5. ACS AI Image: AC2+ACCL+CompilerOpt+CkptOpt: The golden image using AC2 BaseOS with both torch.compile and selective gradient checkpointing optimizations enabled.

Step 1: Log in to the image registry

docker login egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com

Step 2: Enable performance optimizations

• Enable compilation optimization

  Use the transformers Trainer API:

  

• Enable gradient checkpointing

  export CHECKPOINT_OPTIMIZATION=true

Step 3: Start the container

The image has a built-in model training tool named ljperf, which is used to demonstrate how to start a container and run a training task.

# Start the container and enter it.
docker run --rm -it --ipc=host --net=host  --privileged egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/training-xpu-pytorch:[tag]

# Run a training demo.
ljperf --action train --model_name deepspeed/llama3-8b

Known issues in vLLM 0.16.0

• The community version of vLLM 0.16.0 does not support Qwen3.5 series models. PPU provides partial support, but not all features and use cases of the Qwen3.5 models are supported. Potential issues include but are not limited to:

  • Probabilistic failures when the MTP feature is enabled.

  • Precision test failures for bfcl_v3 when the tool call feature is enabled.

• To adapt to GLM-5 and Qwen3.5 series models, the transformers version has been upgraded to 5.2.0. This version does not support Qwen VL models. To run Qwen VL models, you must downgrade the transformers version to 4.57.1.

• DP+EP+DeepEP low latency issues:

  • When you start the server for the first time, you must set export VLLM_ENGINE_READY_TIMEOUT_S=6000. Otherwise, the server will fail to start. The community also encounters this issue on the GU8TF card model.

  • A precision issue exists for the Qwen3 MoE BF16 scenario. This is also a known issue in the community.

• The MiniMax-M2.5 model fails to start when TP=16. Use TP=8 to run int8 quantized weights.

• The MiniMax-M2 model scores low on the ifeval precision test. This issue also occurs on GU8TF instances.

• When the value of --gpu_memory_utilization exceeds 0.96, a segment fault may rarely occur due to a Driver OOM.

• The Flashinfer Sampler may cause a drop in precision. If you encounter this issue, you can use export VLLM_USE_FLASHINFER_SAMPLER=0 to disable the feature.

• The Kimi-linear model has a precision issue, which is also a known issue in the community.

• For Kimi-linear or DeepSeek-OCR series models, using DeepGemm causes the server to fail to start. Set the following environment variables to use the Acext MoE backend.

  # Set environment variables to use the Acext MoE backend.
  export VLLM_USE_DEEP_GEMM=0
  export VLLM_MOE_USE_ACEXT=1
  export VLLM_DENSE_USE_ACEXT=1
  export ACEXT_NUM_TOKENS_LIMIT=16385

• If you use the DeepGemm backend, the server automatically performs a warmup during startup to compile the required DeepGemm kernels. This may lead to a long warmup time:

  • The warmup time is long. We recommend specifying the deepgemm cache path by using an environment variable to avoid repeated compilation: export DG_CACHE_DIR=<your path>.

  • You can use export VLLM_DEEP_GEMM_WARMUP="skip" to skip the DeepGemm warmup. During performance testing, ensure that DeepGemm JIT compilation does not occur. Otherwise, this will cause performance degradation.

• Regarding quantization, SAIL vLLM 0.16.0 is not yet adapted to or optimized for the Marlin kernel. As a result, performance for AWQ (w4a16), GPTQ (w4a16, w8a16), and mxfp4 is suboptimal. We recommend using the int8 w8a8 quantization scheme.