training-xpu-pytorch 26.04

Main Features and Bug Fix Lists

Contents

Key Features and Enhancements

Quick Start

The following example pulls a training-xpu-pytorch image only by using Docker. To use a training-xpu-pytorch image in ACS, you can specify the image address when you create a workload in the console, or specify the image reference in a YAML file.

# Start the container and enter the
docker run --rm -it --ipc=host --net=host  --privileged egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/training-xpu-pytorch:[tag]

# Run the training demo.
ljperf --action train --model_name deepspeed/llama3-8b

Usage suggestions

• The changes in the image involve libraries such as PyTorch and DeepSpeed. Do not reinstall them.

• The zero_optimization.stage3_prefetch_bucket_size in the DeepSpeed configuration is left blank or auto.

• The built-in environment variable NCCL_SOCKET_IFNAME of this image needs to be dynamically adjusted according to the usage scenario:

  • If only 1/2/4/8 cards are applied to a single pod for training or inference tasks, you need to set NCCL_SOCKET_IFNAME=eth0 (the default configuration in this image).

  • When a single pod applies for 16 cards of the whole machine (in this case, you can use the HPN high-speed network) to perform training or inference tasks: You need to set the NCCL_SOCKET_IFNAME=hpn0.

• We recommend that you use this image together with the PPU PIP service provided by Alibaba Cloud to support password-free PIP service in ACS VPCs. You do not need to use other PIP sources in combination. pip config is built into this image. you must configure pip config according to the instructions in this document.

• In the ACS environment to use AcclEP-P (that is, PPU version of DeepEP), you need to set the environment variable export EIC_VSOLAR=1 (this image needs to be set, and it is expected that subsequent images will remove this limit).

• Standard quantization capabilities supported by various frameworks of SDK 2.1.0:

  • vLLM 0.18.0 (on SDK 2.1): per-token/per-channel w8a8(int8)

  • Precautions for quantization capability: Currently, SAIL vLLM has not adapted and optimized the Marlin kernel. The performance of AWQ(w4a16) and GPTQ(w4a16 and w8a16) is poor. Use the w8a8(int8) quantization scheme.

  • A8W8(INT8) quantitative demonstration model.

    • An example of a quantitative model adapted to SDK2.1 is provided. The system logs the ledger and uses the PTG PIP ledger (can be obtained by contacting your account manager):

      • DeepSeek-R1: support per-token/per-channel a8w8(int8) quantization scheme

      • DeepSeek v3.2: support per-token/per-channel a8w8(int8) quantization scheme

      • Kimi-K2-Instruct: support per-token/per-channel a8w8(int8) quantization scheme

      • Qwen3-235B-A22B: support per-token/per-channel a8w8(int8) quantization scheme

      • GLM-5: support per-token/per-channel w8a8(int8) quantization scheme

      • MiniMax-M2.5: support per-token/per-channel w8a8(int8) quantization scheme

      • Qwen3.5-397B-A17B: support per-token/per-channel w8a8(int8) quantization scheme

      • GLM-5.1: support per-token/per-channel w8a8(int8) quantization scheme

• We recommend that you use this image with the latest driver to obtain the best performance. For more information, see Specify a GPU model and driver version for a ACS GPU Pod and GPU driver version description.

Known Issues

• vLLM0.18.0 Image

  • GLM-5 model operation requires an upgrade transformers==5.2.0.

  • DP + EP + DeepEP low latency Question:

    • The first time you start the server, you must set the export VLLM_ENGINE_READY_TIMEOUT_S=6000. Otherwise, the service may fail to start due to the timeout of the DeepGemm WarmUp compiler. The community has the same problem on the GU8TF card type.

    • There is an accuracy issue for the Qwen3 MoE BF16 scenario. The community also has this accuracy problem. Do not use this combination of DP + EP + DeepEP low latency to launch Qwen3 MoE BF16 model weights.

  • If the MiniMax-M2.5 model TP=16 causes service startup failure, use TP=8 to run int8 quantization weights.

  • If the value of the --gpu_memory_utilization is higher than 0.96, the HGGC OOM problem may occur. Lower the gpu_memory_utilization to avoid this problem.

  • If you enable the Flashinfer Sampler feature, the performance can be improved. However, this feature may cause precision loss. By default, this feature is disabled. The community has the same problem. You can turn this feature on or off by VLLM_USE_FLASHINFER_SAMPLER the environment variable.

  • If you use the DeepGemm backend, the warm up is automatically performed during the service startup phase, and the DeepGemm kernel is compiled. This may take a long time to warm up.

    • The warmup time is long. we recommend that you use environment variables to specify the deepgemm cache path export DG_CACHE_DIR=<your path> to avoid repeated compilation.

    • You can skip the DeepGemm warm up export VLLM_DEEP_GEMM_WARMUP="skip". If it is not, the performance will be degraded.

  • In terms of quantization, the current SAIL vLLM version does not adapt and optimize the Marlin kernel. AWQ(w4a16), GPTQ (w4a16, w8a16), and mxfp4 have poor performance. Use the int8 w8a8 quantization scheme.