inference-xpu-pytorch 25.05

Main features and bug fixes

Contents

End-to-end performance evaluation

This evaluation measured the maximum concurrency in vLLM online inference mode while keeping Time to First Token (TTFT) under 3s and Time per Output Token (TPOT) under 100 ms to compare throughput:

• For the DeepSeek-R1-bf16 model, the Output Token Throughput on a single Zhenwu 810E device is 425% of the throughput achieved with the SDK 1.5.0 image.

• For the DeepSeek-R1-w8a8-int8 model, the Output Token Throughput on a single Zhenwu 810E device is 190% of the throughput from the bf16 version.

Quick start

The following example shows how to pull the inference-xpu-pytorch image using Docker and test the inference service with the Qwen2.5-7B-Instruct model.

1. Pull the inference image.

   docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

2. Download the open-source model in ModelScope format.

   pip install modelscope
   cd /mnt
   modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

3. Run the following command to start and enter the container.

   docker run -d -t --network=host --privileged --init --ipc=host \
   --ulimit memlock=-1 --ulimit stack=67108864  \
   -v /mnt/:/mnt/ \
   egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

4. Run an inference test to verify the conversational inference feature of vLLM.

   1. Start the server.

      python3 -m vllm.entrypoints.openai.api_server \
      --model /mnt/Qwen2.5-7B-Instruct \
      --trust-remote-code --disable-custom-all-reduce \
      --tensor-parallel-size 1

   2. Run a test on the client.

      curl http://localhost:8000/v1/chat/completions \
          -H "Content-Type: application/json" \
          -d '{
          "model": "/mnt/Qwen2.5-7B-Instruct",  
          "messages": [
          {"role": "system", "content": "You are a friendly AI assistant."},
          {"role": "user", "content": "Tell me about deep learning."}
          ]}'

      The following output is returned:

      

      For more information about using vLLM, see the vLLM documentation.

Recommendations

• When you run DeepSeek R1 (671B) and Qwen3 models in this image, the supported quantization methods include:

  • vLLM 0.8.5: Supports AWQ (w4a16), GPTQ (w4a16, w8a16), and per-token/per-channel a8w8 (int8) quantization schemes (to run models quantized with AWQ and GPTQ on vLLM 0.8.5, add --quantization moe_wna16).

• This release does not yet support the DeepEP and sailSHMEM features described in PG1 SDK 1.5.1 Release Note. These features will be available in a future release.

• For optimal performance, use this image with driver version 1.5.1. For setup instructions, see Specify the GPU model and driver version for an ACK GPU pod and GPU driver version notes.

Known issues

• The performance of the Qwen3-235B-A22B model will be further optimized in a future release.

• DeepSeek R1 models

  • When you test the DeepSeek model, the --max-model-len setting must be less than 80k. Otherwise, an Out of Memory (OOM) error may occur during runtime.

  • The vLLM image may produce garbled output during inference. We recommend that you disable prefill caching by adding the --no-enable-prefix-caching option.

• Quantization

  • In the SGLang 0.4.6.post1 image, per-token/per-channel a8w8 (int8) quantization is not supported, and the AWQ (w4a16) and GPTQ (w4a16, w8a16) methods are not yet validated.

• Stability

  • An error with mp_distributed_executor may occur during vLLM multi-GPU inference. To prevent this, use VLLM_WORKER_MULTIPROC_METHOD=spawn. For reference, see the GitHub issue at https://github.com/vllm-project/vllm/issues/6152.

  • After multi-GPU inference with vLLM completes, the process may exit unexpectedly. This is a known community issue. For more details, see the open-source community issue at https://github.com/vllm-project/vllm/issues/6145.

• Performance

  • For large MOE models, you can use autotuning tools to improve performance. vLLM supports a performance optimization feature that starts the kernel by using a hardware-specific fused_moe kernel configuration file.