inference-xpu-pytorch-opt
This topic describes the release notes for the inference-xpu-pytorch-opt image.
Before PPU SDK 1.7.0, the SDK-bundled approach for releasing Python packages resulted in long iteration cycles that could not keep pace with rapid community updates. Starting from PPU SDK 1.7.0, the PPU runtime environment decouples Python packages from the PPU SDK. The PPU SDK is now released only for major feature enhancements or updates, ending the monthly release cycle. Agile releases of Python packages now support new models and frameworks from the community.
Due to the change in the PPU SDK release cadence, PPU container images are also no longer released monthly. PPU container images are updated in sync with PPU SDK releases and serve as the base runtime environment for subsequent SDK versions. On the latest base runtime, you can upgrade packages to support new models and frameworks.
Main features and bug fixes
Main features
Upgraded the PPU SDK to 1.7.0, CUDA to 12.9, and the core PyTorch component to 2.8.0.
Provides built-in support for SGLang 0.5.4.post3, including optimizations for expert parallelism and PD separation.
Bug fixes
None
Contents
Image name | inference-xpu-pytorch-opt |
Image tag | sglang0.5.4.post3-250105 |
Use case | Multi-node SGLang inference |
Framework | pytorch |
Requirements | PPU SDK 1.7.0.post1 |
Core components |
|
Image assets
Public image
egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch-opt:sglang0.5.4.post3-250105
Driver requirements
Driver version >= 1.1.0
Key features and enhancements
Expert parallelism and PD separation optimizations
PD separation for model inference separates the Prefill and Decode stages of the inference workload onto different instances. Because the Prefill and Decode stages have different workload characteristics, a PD-separated deployment delivers higher service performance compared to standard open-source frameworks.
Expert parallelism is critical for improving the performance of Mixture-of-Experts (MoE) models in production. During the feed-forward network (FFN) stage, each token is routed to only the top-K experts. This means each expert processes only a subset of tokens, which can lead to low computational efficiency. The core idea of expert parallelism is to distribute the Expert modules (sub-models) across different compute devices based on function or parameters, with each device responsible for the computational tasks of specific experts. This parallelism strategy decouples model size from hardware limitations, enabling models to scale to trillions of parameters. It also uses a dynamic routing mechanism, such as a gating network, to send input data to the most appropriate experts, which increases computational efficiency.
This feature is available only for the inference-nv-pytorch-opt image. To use this feature, contact your BTE to develop a solution.
ACCL communication library
ACCL is a high-performance network communication library developed by Alibaba for Lingjun products. It is available in three versions: ACCL-N for NVIDIA GPUs, ACCL-P for PPUs, and ACCL-R for AMD accelerators. ACCL-N is based on NVIDIA NCCL, offering full compatibility and including bug fixes and optimizations for performance and stability. ACCL-P is a collective communication library built upon T-Head's open-source pccl. ACCL-R is built upon AMD's open-source ROCm rccl. This release ports key features from ACCL and ACCL-N to pccl, resolves known issues, and is deeply customized for integration with Alibaba Cloud's proprietary networking components.
E2E performance evaluation
To obtain PerfHub performance data, contact your BTE.
Quick start
PD separation
This image supports PD-separated deployment. We recommend contacting your BTE to develop a solution. For more information, see ACS PD-separated Deployment of MoE Models (EP Optimization) Best Practices.
Non-PD separation
The following example shows how to pull the inference-xpu-pytorch-opt image using Docker and test an inference service with the Qwen2.5-7B-Instruct model.
To use the inference-xpu-pytorch-opt image in ACS, you can specify the image address when creating a workload in the console, or reference the image in a YAML file.
For guidance on using XPU large model inference images in an ACS environment, see Guidance on Using LLM Inference Images in ACS Cluster Mode.
For instructions on deploying the DeepSeek inference service in an ACS environment, see Using PPU to Quickly Deploy DeepSeek V3/R1 Inference Service in ACS.
Pull the inference container image.
NoteBefore you can pull ACS AI container images over the public network, you must obtain Authentication for PPU AI container images. We recommend that you Use VPC to accelerate AI container image pulling to reduce the image pull time.
docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch-opt:[tag]Download the open-source model from ModelScope.
pip install modelscope cd /mnt modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-InstructRun the following command to start the container.
docker run -d -t --network=host --privileged --init --ipc=host \ --ulimit memlock=-1 --ulimit stack=67108864 \ -v /mnt/:/mnt/ \ egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch-opt:[tag]Run an inference test to validate the chat feature.
Start the server.
python3 -m sglang.launch_server \ --model-path /mnt/Qwen2.5-7B-Instruct \ --tensor-parallel-size 1Run a test on the client.
curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "/mnt/Qwen2.5-7B-Instruct", "messages": [ {"role": "system", "content": "You are a friendly AI assistant."}, {"role": "user", "content": "Tell me about deep learning."} ]}'The command returns the following output:
Usage recommendations
The image includes modified libraries such as PyTorch. Do not reinstall them.
Standard quantization capabilities supported by PPU SDK 1.7.0 for various frameworks:
The following are examples of quantized models compatible with PPU SDK 1.7.0. Use your PTG pip service credentials to log in. If you do not have an account, contact your BTE to obtain one.
DeepSeek-R1: supports per-token/per-channel a8w8 (int8) quantization scheme
DeepSeek v3.2: supports per-token/per-channel a8w8 (int8) quantization scheme
Kimi-K2-Instruct: supports per-token/per-channel a8w8 (int8) quantization scheme
Qwen3-235B-A22B: supports per-token/per-channel a8w8 (int8) quantization scheme
For optimal performance, we recommend using this image with driver version 1.5.1 or later. For setup instructions, see Specify GPU Model and Driver Version for an ACS GPU Pod and GPU Driver Version Notes.
When using AcclEP-P (the PPU version of DeepEP) in an ACS environment, you must set the environment variable
export EIC_VSOLAR=1. This setting is required for the current image but we expect to remove it in future image releases.The built-in environment variable
NCCL_SOCKET_IFNAMEin this image must be adjusted based on your use case:When a single Pod requests 1, 2, 4, or 8 GPUs for a training or inference task: Set
NCCL_SOCKET_IFNAME=eth0(the default configuration in this image).When a single Pod requests all 16 GPUs on a full instance for a training or inference task (which allows you to use HPN): Set
NCCL_SOCKET_IFNAME=hpn0.
We recommend using this image with the PPU pip service provided by Alibaba Cloud in ACS products. This allows you to use the pip service within an ACS VPC without requiring credentials or adding other pip sources. This image has a built-in pip configuration. However, you may still need additional configuration based on the guide and your specific use case.