inference-xpu-pytorch 25.11

Main features

• For the 25.11-v1.7.0-vllm0.10.2-torch2.8-cu129-20251113 and 25.11-v1.7.0-sglang0.5.2-torch2.8-cu129-20251113 images:

  • Upgraded PPU SDK to 1.7.0 and CUDA to 12.9.

  • Upgraded PyTorch to 2.8.0.

  • Upgraded vLLM to 0.10.2. To run Qwen3-VL or DeepSeek V3.2 models, you must manually upgrade vLLM to version 0.11.0.

  • Upgraded SGLang to 0.5.2, FlashInfer to 0.3.1, and Flash Attention to 2.8.2. To run the Qwen3-VL model, you must manually upgrade SGLang to version 0.5.3 or manually upgrade SGLang to version 0.5.5.

• For the 25.11-v1.7.0.post1-sglang0.5.5-torch2.8-cu129-20251216 image:

  • Upgraded PPU SDK to V1.7.0_hotfix.

  • Upgraded SGLang to 0.5.5 and FlashInfer to 0.4.0rc3.

• For the 25.11-v1.7.0.post1-vllm0.11.1-torch2.8-cu129-20260105 image:

  • Upgraded PPU SDK to V1.7.0_hotfix.

  • Upgraded vLLM to 0.11.1.

  • Upgraded transformers to 4.57.0.

Bug fixes

• The 25.11-v1.7.0.post1-sglang0.5.5-torch2.8-cu129-20251216 image fixes a crash that occurred in multi-node scenarios. An incorrect orchestration of proxy parameters (sub args) during the fusion of multiple PCCL P2P operations caused the issue.

Public images

• egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.11-v1.7.0-vllm0.10.2-torch2.8-cu129-20251113

• egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.11-v1.7.0-sglang0.5.2-torch2.8-cu129-20251113

• egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.11-v1.7.0.post1-sglang0.5.5-torch2.8-cu129-20251216

• egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.11-v1.7.0.post1-vllm0.11.1-torch2.8-cu129-20260105

VPC images

Recommendations

• Standard quantization capabilities supported by each framework in SDK 1.7.0:

  • vLLM 0.10.2 (built-in) or 0.11.0 (upgraded via pip): Supports per-token/per-channel a8w8 (int8), AWQ (w4a16), and GPTQ (w4a16, w8a16) quantization schemes.

  • SGLang 0.5.2 (built-in), 0.5.3 (upgraded via pip), or 0.5.5 (upgraded via pip): Supports per-token/per-channel a8w8 (int8), AWQ (w4a16), and GPTQ (w4a16, w8a16) quantization schemes.

    To run an a8w8 (int8) quantized model, add the --quantization w8a8_int8 option.

  • The following quantized models are compatible with SDK 1.7.0. System login credentials reuse the PTG PIP credentials (contact your PDSA for access):

    • DeepSeek-R1: Supports per-token/per-channel a8w8 (int8) quantization.

    • DeepSeek V3.2: Supports per-token/per-channel a8w8 (int8) quantization.

    • Kimi-K2-Instruct: Supports per-token/per-channel a8w8 (int8) quantization.

    • Qwen3-235B-A22B: Supports per-token/per-channel a8w8 (int8) quantization.

• For optimal performance, we recommend using this image with driver version 1.5.1 or later. For configuration instructions, see Specify GPU models and driver versions for ACS GPU Pods and GPU driver version notes.

• To use AcclEP-P (the PPU version of DeepEP) in an ACS environment, you must set the environment variable export EIC_VSOLAR=1. This setting is required for the current image, but we plan to remove this limitation in a future release.

• You must adjust the built-in NCCL_SOCKET_IFNAME environment variable based on your use case:

  • When a single Pod requests 1, 2, 4, or 8 cards for an inference task, set NCCL_SOCKET_IFNAME=eth0 (the default configuration in this inference image).

  • When a single Pod requests all 16 cards on an entire machine for an inference task (which allows you to use HPN), set NCCL_SOCKET_IFNAME=hpn0.

• We recommend using this image with the Alibaba Cloud PPU PIP service in ACS. It provides one-stop, password-free access to the PIP service within an ACS VPC, eliminating the need to use other PIP sources. The image includes the required pip configuration, but you must still configure it for your specific use case as described in the documentation.

Known issues

• vLLM 0.10.2 performance issue

  For Qwen models, performance has regressed compared to previous versions because fused_topk in the fused Mixture-of-Experts (MoE) layer is not wrapped with torch.compile. A patch has been submitted to the open-source community and is slated for inclusion in vLLM 0.11.1. For more information, see the GitHub community content.

• In a two-node vLLM scenario, if the container image has transformers version 4.56.0 or later and torchao is also installed, the run fails. The same issue exists on GPUs. Uninstalling torchao is a valid workaround.

• SGLang

  • DeepGemm compiles at runtime. Ensure a sufficient warm-up period when you benchmark performance.

  • DeepSeek V3.2 currently requires dp-attention to be enabled and does not yet support tensor parallelism (TP).

  • Qwen-235B-A22B shows a performance regression with short outputs (output_len=400).

  • In the DeepSeek-R1 SLA test scenario with an input length of 900 and an output length of 400, the maximum concurrency does not achieve maximum throughput.

• In the DeepSeek-R1 PD separation scenario, SGLang 0.5.5 performance regresses by 20% compared to SGLang 0.5.2. This regression occurs because the SGLang community removed support for the load_balance_method = "minimum_tokens" option in version 0.5.3 and later. In theory, H20 also experiences this performance regression. For more information, see the GitHub community PR.

Appendix 1: Upgrade vLLM to 0.11.0

pip uninstall vllm -y
pip install vllm==0.11.0+ppu1.7.0 
# Do not use numpy 2.x
pip install numpy==1.26.4
# Upgrade transformers
pip install transformers==4.57.0

# Optional step if running the DeepSeek v3.2 model
# pip install scikit-learn==1.7.1

pip list | grep vllm
pip list | grep transformers
pip list | grep numpy

Appendix 2: Upgrade SGLang to 0.5.3

# Uninstall sglang and sgl-kernel 
pip uninstall -y sglang sgl-kernel 
# Install sgl-kernel
pip install sgl_kernel==0.3.14.post1+ppu1.7.0 --force-reinstall
# Install sglang
pip install sglang==0.5.3+ppu1.7.0 --no-deps --force-reinstall

# Install transformers 
pip install -U transformers==4.57.1 openai_harmony --force-reinstall
# Install fast_hadamard_transform and numpy 
pip install fast_hadamard_transform numpy==1.26.0

# Optional: Upgrade flashinfer. If you do not upgrade flashinfer, you can use the Triton backend by adding the --attention-backend triton option to the command.   
# pip install flashinfer_python==0.4.0rc3+ppu1.7.0.oe --force-reinstall --no-deps

pip list | grep sgl
pip list | grep transformer

Appendix 3: Upgrade SGLang to 0.5.5

# Upgrade SGLang
## Uninstall sgl-kernel, sglang-router, and sglang
pip uninstall -y sgl-kernel sglang-router sglang
pip install sgl-kernel==0.3.16.post5+ppu1.7.0.post1 --force-reinstall --no-deps
pip install sglang-router==0.2.2+ppu1.7.0.post1 --force-reinstall --no-deps
pip install sglang==0.5.5+ppu1.7.0.post1 --force-reinstall --no-deps

# Upgrade DeepGemm
pip install deep_gemm==1.0.0+ppu1.7.0.post1 --force-reinstall --no-deps

# Upgrade DeepEP
pip install deep_ep==1.0.0+ppu1.7.0.post1 --force-reinstall --no-deps

# Upgrade Flash Attention 2.8.2
pip install flash_attn==2.8.2+ppu1.7.0.post1 --force-reinstall --no-deps
pip install flash_attn_3==2.8.2+ppu1.7.0.post1 --force-reinstall --no-deps

# Upgrade FlashMLA
pip install flash_mla==1.0.0+ppu1.7.0.post1 --force-reinstall --no-deps

# Upgrade xgrammar and transformers
pip install -U xgrammar==0.1.25 transformers==4.57.1 numpy==1.26.0 --force-reinstall --no-deps

# Upgrade FlashInfer
pip install flashinfer_python==0.4.0rc3+ppu1.7.0.oe --force-reinstall --no-deps

pip list | grep sglang
# Expected output
sglang                            0.5.5+ppu1.7.0.post1
sglang-router                     0.2.2+ppu1.7.0.post1

pip list | grep flash
# Expected output
flash-attn                        2.8.2+ppu1.7.0.post1
flash-attn-3                      2.8.2+ppu1.7.0.post1
flash_mla                         1.0.0+ppu1.7.0.post1
flashinfer-python                 0.4.0rc3+ppu1.7.0.oe

pip list | grep transformers
# Expected output
transformers                      4.57.1