inference-xpu-pytorch 25.06

Main features and bug fixes

Contents

Images

We recommend using a VPC endpoint to accelerate AI container image pulls and reduce pull times.

E2E performance evaluation

In vLLM online inference mode, we measured the maximum concurrency while keeping the Time to First Token (TTFT) under 3s and Time Per Output Token (TPOT) under 100ms, and then compared the throughput.

• vllm0.8.5:

  • The single-node Output Token Throughput for the Qwen3-235B-A22B (bf16) model is 132% of that of the v1.5.1 image.

  • The single-node Output Token Throughput for the Qwen3-235B-A22B-INT8 model is 122% of that of the bf16 model.

Quick start

The following example shows how to pull the inference-xpu-pytorch image using Docker and test the inference service with the Qwen2.5-7B-Instruct model.

1. Pull the inference image.

   docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

2. Download the open-source model in ModelScope format.

   pip install modelscope
   cd /mnt
   modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

3. Run the following command to start and enter the container.

   docker run -d -t --network=host --privileged --init --ipc=host \
   --ulimit memlock=-1 --ulimit stack=67108864  \
   -v /mnt/:/mnt/ \
   egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

4. Run an inference test to verify the conversational inference feature of vLLM.

   1. Start the server.

      python3 -m vllm.entrypoints.openai.api_server \
      --model /mnt/Qwen2.5-7B-Instruct \
      --trust-remote-code --disable-custom-all-reduce \
      --tensor-parallel-size 1

   2. Run a test on the client.

      curl http://localhost:8000/v1/chat/completions \
          -H "Content-Type: application/json" \
          -d '{
          "model": "/mnt/Qwen2.5-7B-Instruct",  
          "messages": [
          {"role": "system", "content": "You are a friendly AI assistant."},
          {"role": "user", "content": "Tell me about deep learning."}
          ]}'

      The following output is returned:

      

      For more information about using vLLM, see the vLLM documentation.

Usage notes

• When you use this image to run DeepSeek R1 (671B) and Qwen3 quantized models, the following quantization methods are supported:

  • DeepSeek R1 (671B):

    • VLLM 0.8.5: Supports AWQ (w4a16), GPTQ (w4a16, w8a16), and per-token/per-channel a8w8 (int8) quantization.

    • SGLang 0.4.6.post1: Does not support AWQ (w4a16) and GPTQ (w4a16, w8a16) quantization due to a known issue in the open source community. It supports per-token/per-channel A8W8 (int8) quantization. To run an A8W8 (int8) quantized model, add the --quantization w8a8_int8 option.

  • Qwen3:

    • VLLM 0.8.5: Supports AWQ (w4a16), GPTQ (w4a16, w8a16), and per-token/per-channel a8w8 (int8) quantization.

    • SGLang 0.4.6.post1: Supports AWQ (w4a16), GPTQ (w4a16, w8a16), and per-token/per-channel A8W8 (int8) quantization. To run a quantized model using AWQ or GPTQ on SGLang 0.4.6.post1, add the --quantization moe_wna16 option. To run an A8W8 (int8) quantized model, add the --quantization w8a8_int8 option.

• The DeepEP and sailSHMEM features mentioned in the PPU SDK 1.5.1 and 1.5.2 release notes are not yet supported. A timeline for support has not been determined.

• For optimal performance, we recommend using this image with driver version 1.5.1 or later. For setup instructions, see Specify GPU Model and Driver Version for ACS GPU Pods and GPU Driver Version Information.

• You must dynamically adjust the built-in NCCL_SOCKET_IFNAME environment variable in this image based on the use case:

  • If a single pod requests 1, 2, 4, or 8 cards for an inference task, set NCCL_SOCKET_IFNAME=eth0. This is the default configuration in this inference image.

  • If a single pod requests all 16 cards on a full machine for an inference task (which allows you to use HPN), set NCCL_SOCKET_IFNAME=hpn0.

Known issues

• Qwen3 models:

  • The Qwen3-235B-A22B model has an inference accuracy issue. As a workaround, set the ACEXT_ENABLE_MOE_GEMV=0 environment variable. This issue will be fixed in a future release.

• Qwen2.5 models:

  • When you run the Qwen2.5-omni model with the vLLM image, a ModuleNotFoundError: No module named 'vllm.vllm_flash_attn.layers'  error occurs. This is triggered by a known issue in the vLLM 0.8.5 open source community and will be fixed in a future vLLM update.

• DeepSeek R1 models:

  • When you test the DeepSeek R1 model, the --max-model-len setting must be less than 80k to prevent potential out-of-memory (OOM) errors during runtime.

• Stability issues:

  • When you perform multi-card inference with vLLM, an error related to mp_distributed_executor may occur. To prevent this, set VLLM_WORKER_MULTIPROC_METHOD=spawn. For more information, see this discussion in the open source community.

  • After multi-card inference with vLLM, a process may unexpectedly exit with an error. This is a known issue in the open source community. For more information, see this discussion in the open source community.

• Performance issues:

  • For MoE models, you can use autotuning tools to improve performance. vLLM supports a performance optimization feature that allows you to launch kernels by using hardware-specific fused_moe kernel configuration files.