inference-xpu-pytorch 25.01

Release highlights

New features

• Upgraded the PPU SDK in the base image to v1.4.1.

• Upgraded vLLM to v0.6.4.post1.

• Upgraded Torch to 2.5.1.

Bug fixes

This release does not include any bug fixes.

Image assets

Quick start

The following example shows how to pull the inference-xpu-pytorch image using Docker and test the inference service with the Qwen2.5-7B-Instruct model.

1. Pull the inference image.

   docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

2. Download the open-source model in ModelScope format.

   pip install modelscope
   cd /mnt
   modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

3. Run the following command to start and enter the container.

   docker run -d -t --network=host --privileged --init --ipc=host \
   --ulimit memlock=-1 --ulimit stack=67108864  \
   -v /mnt/:/mnt/ \
   egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

4. Run an inference test to verify the conversational inference feature of vLLM.

   1. Start the server.

      python3 -m vllm.entrypoints.openai.api_server \
      --model /mnt/Qwen2.5-7B-Instruct \
      --trust-remote-code --disable-custom-all-reduce \
      --tensor-parallel-size 1

   2. Run a test on the client.

      curl http://localhost:8000/v1/chat/completions \
          -H "Content-Type: application/json" \
          -d '{
          "model": "/mnt/Qwen2.5-7B-Instruct",  
          "messages": [
          {"role": "system", "content": "You are a friendly AI assistant."},
          {"role": "user", "content": "Tell me about deep learning."}
          ]}'

      The following output is returned:

      

      For more information about using vLLM, see the vLLM documentation.

Known issues

• Performance issues

  • The Autotune tool improves the performance of Mixture-of-experts (MoE) large models. For further optimization, vLLM can start its kernel with a hardware-specific fused_moe kernel configuration file.

• Quantization issues

  • By default, vLLM uses the Marlin kernel to accelerate inference for GPTQ, AWQ, W8A8, and WOQ quantization. You can explicitly specify a quantization method to use vLLM's original quantization kernels, such as Marlin, W8A8, AWQ, or GPTQ. For example, you can specify gptq_acext or awq_acext for quantized inference. However, acext does not currently support quantization with act_order=True or GPTQ-Int8 quantized weights. This will be addressed in a future release.

  • vLLM uses vllm-flash-attn, xformers, or flashinfer as the attention backend. The default vllm-flash-attn backend does not yet outperform the native GPU implementation. The next release will integrate A8W8 quantization in vLLM with the acext library, though its performance will require further optimization.

  • FP8 quantization is currently very slow because it has not been adapted or optimized for PPU. A support plan for this feature is not yet available.

• Stability issues

  • This is a known community issue where vLLM processes may exit unexpectedly after a multi-GPU inference task completes. For more details, see the issue in the open source community.