inference-xpu-pytorch 25.09-ZHENWU PPU Cloud Service(ppu)-阿里云帮助中心

This document provides the release notes for inference-xpu-pytorch 25.09.

Main features and bug fixes

Main features

Upgraded the PPU SDK in the base image to 1.6.1.
Upgraded the vLLM version in the vLLM image to 0.10.0.
Upgraded the SGLang version in the SGLang image to 0.4.10.
Upgraded FlashInfer to 0.2.9rc2.
Upgraded Flash Attention to 2.7.4.post1 with support for the FA3 feature.

Bug fixes

None

	inference-xpu-pytorch	inference-xpu-pytorch
Image tag	25.09-v1.6.1-vllm0.10.0-torch2.7-cu128-20250924	25.09-v1.6.1-sglang0.4.10-torch2.7-cu128-20250924
Application scenario	large model inference	large model inference
Framework	pytorch	pytorch
Requirements	PPU SDK V1.6.1	PPU SDK V1.6.1
System components	Ubuntu 24.04 Python 3.12 Torch 2.7.0 CUDA 12.8 ACCL-P 1.6.0-10 AcclEP-P 1.0.0.5+0e3ede0 EIC 1.3.8.2 eic-sailshmem 1.0.3.1.0.gda3c51b accelerate 1.10.1 apex 0.1 deepseed 0.14.4 diffusers 0.35.1 flash-attn 2.7.4.post1 flashinfer-python 0.2.6.post1 imageio 2.37.0 imageio-ffmpeg 0.6.0 mooncake-transfer-engine 0.3.4.post2 peft 0.12.0 ray 2.49.2 xformers 0.0.30 transformers 4.53.3 transformer_engine 2.3.0+eb92009 triton 3.3.0+gitd63d56cc torchao 0.11.0 torchvision 0.22.0 vllm 0.10.0 xgrammar 0.1.21 xfuser 0.4.4 ljperf 0.1.0+477686c5	Ubuntu 24.04 Python 3.12 Torch 2.7.0 CUDA 12.8 ACCL-P 1.6.0-10 AcclEP-P 1.0.0.5+0e3ede0 EIC 1.3.8.2 eic-sailshmem 1.0.3.1.0.gda3c51b accelerate 1.10.0 diffusers 0.35.1 decord 0.6.0 deep_ep 1.6.0+1242c66 deep_gemm 1.0.0+82eea9c deepspeed 0.14.4 flash-attn 2.8.2 flash-attn-3 3.0.0b1 flash_mla 1.0.0 flashinfer-python 0.2.9rc2 imageio 2.37.0 imageio-ffmpeg 0.6.0 mooncake-transfer-engine 0.3.4.post2 peft 0.12.0 ray 2.49.2 sglang 0.4.10 sgl-kernel 0.2.8 transformers 4.51.3 transformer_engine 2.3.0+eb92009 torchao 0.11.0 triton 3.3.0+gitd63d56cc vllm 0.9.1 xformers 0.0.29.post1 xgrammar 0.1.19 xfuser 0.4.4 ljperf 0.1.0+477686c5

Image assets

We recommend that you accelerate AI container image pulls over VPC to reduce pull times.

Public image

egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.09-v1.6.1-vllm0.10.0-torch2.7-cu128-20250924
egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.09-v1.6.1-sglang0.4.10-torch2.7-cu128-20250924

VPC images

acs-registry-vpc.{region-id}.cr.aliyuncs.com/egslingjun/{image:tag}
{region-id} is the ID of the region where your Container Service for Kubernetes (ACK) cluster is located, such as cn-beijing or cn-wulanchabu.
{image:tag} is the actual image name and tag.

E2E performance evaluation

In vLLM online inference mode, we tested the maximum concurrency and compared throughput while keeping the time to first token (TTFT) under 3 seconds and the time per output token (TPOT) under 100 ms. The input and output token lengths were set to 4096 and 2048, respectively.

The single-node output token throughput of the DeepSeek-R1-bf16 and DeepSeek-R1-W8A8 models is comparable to version 25.08.

Quick start

The following example shows how to pull the inference-xpu-pytorch image using Docker and test the inference service with the Qwen2.5-7B-Instruct model.

Note

To use the inference-xpu-pytorch image in Container Service for Kubernetes (ACK), you can select it from the artifact center page when you create a workload in the Console, or specify the image reference in a YAML file.

For instructions on using XPU large model inference images in an ACK environment, see Guide to using LLM large model inference images in ACK clusters.
For instructions on deploying the DeepSeek inference service in an ACK environment, see Quickly deploy DeepSeek V3/R1 inference service on PPU in ACK.

Pull the inference image.

docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

Download the open-source model in ModelScope format.

pip install modelscope
cd /mnt
modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

Run the following command to start and enter the container.

docker run -d -t --network=host --privileged --init --ipc=host \
--ulimit memlock=-1 --ulimit stack=67108864  \
-v /mnt/:/mnt/ \
egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

Run an inference test to verify the conversational inference feature of vLLM.

Start the server.

python3 -m vllm.entrypoints.openai.api_server \
--model /mnt/Qwen2.5-7B-Instruct \
--trust-remote-code --disable-custom-all-reduce \
--tensor-parallel-size 1

Run a test on the client.

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "/mnt/Qwen2.5-7B-Instruct",  
    "messages": [
    {"role": "system", "content": "You are a friendly AI assistant."},
    {"role": "user", "content": "Tell me about deep learning."}
    ]}'

The following output is returned:

For more information about using vLLM, see the vLLM documentation.

Usage recommendations

DeepSeek R1 (671B):
- vLLM 0.10.0: Supports per-token/per-channel a8w8 (int8), AWQ (w4a16), and GPTQ (w4a16, w8a16) quantization schemes.
- SGLang 0.4.10: Supports per-token/per-channel A8W8 (int8), AWQ (w4a16), and GPTQ (w4a16, w8a16) quantization schemes. To run an A8W8 (int8) quantized model, you must add the --quantization w8a8_int8 option.
Qwen3:
- vLLM 0.10.0: Supports per-token/per-channel a8w8 (int8), AWQ (w4a16), and GPTQ (w4a16, w8a16) quantization schemes.
- SGLang 0.4.10: Supports per-token/per-channel A8W8 (int8), AWQ (w4a16), and GPTQ (w4a16, w8a16) quantization schemes. To run an A8W8 (int8) quantized model, you must add the --quantization w8a8_int8 option.
For optimal performance, we recommend using this image with driver version 1.5.1 or later. For setup instructions, see Specify GPU Models and Driver Versions for ACS GPU Pods and GPU Driver Version Description.
Adjust the built-in environment variable NCCL_SOCKET_IFNAME in this image based on your use case:
- When a single pod requests 1, 2, 4, or 8 GPUs for an inference task, set NCCL_SOCKET_IFNAME=eth0. This is the default configuration in this inference image.
- When a single pod requests all 16 GPUs on a node for an inference task (which allows you to use HPN), set NCCL_SOCKET_IFNAME=hpn0.
We recommend using this image with the PPU PIP service provided by Alibaba Cloud. This service provides one-stop, password-free access to PIP within an ACS VPC, eliminating the need to use other PIP sources. This image includes the necessary pip configuration, but you still need to complete the configuration for your use case as described in the documentation.

Known issues

In some scenarios with SGLang 0.4.10 on Qwen3-8B, throughput under high concurrency is lower than under low concurrency. This is because high concurrency can hit tile configurations where the gemm and FlashInfer kernels perform poorly. In such cases, we recommend using a lower concurrency setting that provides better throughput.
SGLang 0.4.10 uses the FA3 backend by default. In scenarios with Qwen3-235B-A22B, 4,000 input tokens, and 1,500 output tokens, the FA3 backend performs worse than FlashInfer. We recommend using the FlashInfer backend.
SGLang 0.4.10 uses the FA3 backend by default. In BF16 scenarios with Qwen3-235B-A22B, the process may hang during the CUDA graph capturing phase. To resolve this, try lowering the --mem_fraction_static setting, for example, to 0.8.
In a vLLM dual-node setup, the run fails if the container image has transformers >= 4.53.3 and torchao installed. This issue also occurs on GPUs. To work around this, uninstall torchao.
To use AcclEP-P (the PPU version of DeepEP) in an ACS environment, run the command export EIC_VSOLAR=1. This is a requirement for the current image and is expected to be removed in future images.