inference-xpu-pytorch 25.04, inference-xpu-pytorch 25.04

Key features and bug fixes

Image contents

Image references

Quick start

The following example shows how to pull the inference-xpu-pytorch image using Docker and test the inference service with the Qwen2.5-7B-Instruct model.

1. Pull the inference image.

   docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

2. Download the open-source model in ModelScope format.

   pip install modelscope
   cd /mnt
   modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

3. Run the following command to start and enter the container.

   docker run -d -t --network=host --privileged --init --ipc=host \
   --ulimit memlock=-1 --ulimit stack=67108864  \
   -v /mnt/:/mnt/ \
   egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

4. Run an inference test to verify the conversational inference feature of vLLM.

   1. Start the server.

      python3 -m vllm.entrypoints.openai.api_server \
      --model /mnt/Qwen2.5-7B-Instruct \
      --trust-remote-code --disable-custom-all-reduce \
      --tensor-parallel-size 1

   2. Run a test on the client.

      curl http://localhost:8000/v1/chat/completions \
          -H "Content-Type: application/json" \
          -d '{
          "model": "/mnt/Qwen2.5-7B-Instruct",  
          "messages": [
          {"role": "system", "content": "You are a friendly AI assistant."},
          {"role": "user", "content": "Tell me about deep learning."}
          ]}'

      The following output is returned:

      

      For more information about using vLLM, see the vLLM documentation.

Known issues

• DeepSeek

  • When testing the DeepSeek model, the --max-model-len setting must be < 80k. Otherwise, you may encounter an out-of-memory (OOM) error at runtime.

  • Due to a known community issue, SGLang does not support GPTQ-quantized models.

• Performance issues

  • For mixture-of-experts (MoE) models, using the Autotune tool can improve performance. vLLM can improve performance by using hardware-specific fused_moe kernel configuration files.

• Quantization issues

  • By default, the LLM uses the Marlin Kernel to accelerate inference for models quantized with GPTQ, AWQ, W8A8, or WOQ. Alternatively, you can use vLLM's original quantization inference kernels by explicitly setting the quantization parameter. For example, you can specify values such as marlin, w8a8, awq, gptq, gptq_acext, or awq_acext. However, the acext kernels currently do not support quantization with act_order=True or GPTQ-Int8 quantized weights. This will be optimized in a future version.

  • vLLM FP8 quantization depends on the SM89/90 architecture and has not been adapted or optimized for PPUs. A support plan for PPUs is not yet available.

  • VLLM dense A8W8 quantization integrates with the acext library, and performance requires further optimization. MoE W8A8 quantization is not currently supported and is planned for the next release.

  • You can load DPSK FP8 weights and perform W8A16 quantization inference. The performance is comparable to BF16.

  • vLLM uses vllm-flash-attn as the default attention backend. Its performance on PPU does not yet match that of the GPU version. This will be improved in a future release.

• Stability issues

  • When performing multi-card inference with VLLM, an error related to mp_distributed_executor may occur. To avoid this issue, use VLLM_WORKER_MULTIPROC_METHOD=spawn. For more information, see the related issue in the open-source community: [Bug]: When tensor_parallel_size>1, RuntimeError: Cannot re-initialize CUDA in forked subprocess. #6152.

  • A known community issue may cause a process to exit randomly after vLLM multi-card inference completes. For details, see the open-source community issue [Bug]: When using tp for inference, an error occurs: Worker VllmWorkerProcess pid 3283517 died, exit code: -15. #6145.