inference-xpu-pytorch 25.08

Main features and bug fixes

Contents

Image assets

End-to-end performance benchmarks

This benchmark measured the maximum concurrency and throughput in vLLM online inference mode, while maintaining a Time to First Token (TTFT) under 3s and a Time Per Output Token (TPOT) under 100 ms.

• vllm 0.9.1:

  • The single-node Output Token Throughput for the DeepSeek-R1-bf16 model, which degraded in version 25.07, has been restored in this release.

  • The single-node output token throughput for the DeepSeek-R1-W8A8 model is 197% of that of the bf16 model.

Quick start

The following example shows how to pull the inference-xpu-pytorch image using Docker and test the inference service with the Qwen2.5-7B-Instruct model.

1. Pull the inference image.

   docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

2. Download the open-source model in ModelScope format.

   pip install modelscope
   cd /mnt
   modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

3. Run the following command to start and enter the container.

   docker run -d -t --network=host --privileged --init --ipc=host \
   --ulimit memlock=-1 --ulimit stack=67108864  \
   -v /mnt/:/mnt/ \
   egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

4. Run an inference test to verify the conversational inference feature of vLLM.

   1. Start the server.

      python3 -m vllm.entrypoints.openai.api_server \
      --model /mnt/Qwen2.5-7B-Instruct \
      --trust-remote-code --disable-custom-all-reduce \
      --tensor-parallel-size 1

   2. Run a test on the client.

      curl http://localhost:8000/v1/chat/completions \
          -H "Content-Type: application/json" \
          -d '{
          "model": "/mnt/Qwen2.5-7B-Instruct",  
          "messages": [
          {"role": "system", "content": "You are a friendly AI assistant."},
          {"role": "user", "content": "Tell me about deep learning."}
          ]}'

      The following output is returned:

      

      For more information about using vLLM, see the vLLM documentation.

Usage notes

Known issues

• vLLM stability issues:

  • During multi-GPU inference, you may encounter an mp_distributed_executor error. To work around this issue, set VLLM_WORKER_MULTIPROC_METHOD=spawn. See the open source community.

  • After multi-GPU inference completes, vLLM may randomly exit with an error. This is a known upstream issue. See the open source community for details.

• vLLM quantization issues:

  • vLLM uses the Marlin kernel by default to accelerate GPTQ/AWQ quantized inference. You can explicitly specify quantization gptq_acext or quantization awq_acext to use acext for quantized inference instead. However, acext does not currently support quantization with act_order=True or GPTQ-Int8 weights.

  • vLLM FP8 quantization requires an SM89/90 architecture, which PPU does not natively support. vLLM 0.9.1 provides an FP8 Marlin implementation (which performs W8A16 quantized inference) that allows FP8 models to run on architectures with SM versions lower than 89. While PPU can run FP8 models this way, it has not yet been adapted or optimized for this purpose.

• vLLM-v0.9.1 estimates less available KV cache space than v0.8.5. This reduces concurrency and degrades performance in scenarios constrained by the KV cache, such as running DeepSeek-R1-bf16 on a single-node Zhenwu 810E.

  • Use the A8W8 quantization model on Zhenwu 810E to avoid this performance degradation.

• By default, SGLang-v0.4.7 uses shared experts when running the DeepSeek-R1 / DeepSeek-V3 A8W8-INT8 model, while SGLang-v0.4.6.post1 does not. This causes SGLang-v0.4.7 to have lower performance than SGLang-v0.4.6.post1. To disable shared experts, use the --disable-shared-experts-fusion option. With this option, the performance of SGLang-v0.4.7 is comparable to that of SGLang-v0.4.6.post1.

• Setting the --mem-fraction-static option to a large value for the SGLang-v0.4.7 image causes insufficient single-node GPU memory, resulting in a decode out of memory error. This is a known community issue.

• The following flashinfer modules have known precision issues that cause unit tests to fail.

  1. PoD Attention.

  2. Segment gemm.