inference-xpu-pytorch 25.02

Key features and bug fixes

Image assets

Quick start

The following example shows how to pull the inference-xpu-pytorch image using Docker and test the inference service with the Qwen2.5-7B-Instruct model.

1. Pull the inference image.

   docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

2. Download the open-source model in ModelScope format.

   pip install modelscope
   cd /mnt
   modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

3. Run the following command to start and enter the container.

   docker run -d -t --network=host --privileged --init --ipc=host \
   --ulimit memlock=-1 --ulimit stack=67108864  \
   -v /mnt/:/mnt/ \
   egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

4. Run an inference test to verify the conversational inference feature of vLLM.

   1. Start the server.

      python3 -m vllm.entrypoints.openai.api_server \
      --model /mnt/Qwen2.5-7B-Instruct \
      --trust-remote-code --disable-custom-all-reduce \
      --tensor-parallel-size 1

   2. Run a test on the client.

      curl http://localhost:8000/v1/chat/completions \
          -H "Content-Type: application/json" \
          -d '{
          "model": "/mnt/Qwen2.5-7B-Instruct",  
          "messages": [
          {"role": "system", "content": "You are a friendly AI assistant."},
          {"role": "user", "content": "Tell me about deep learning."}
          ]}'

      The following output is returned:

      

      For more information about using vLLM, see the vLLM documentation.

Known issues

• DeepSeek

  • To avoid out-of-memory (OOM) errors when testing DeepSeek models, you must set the --max-model-len parameter to a value less than 80,000.

  • SGLang does not support models that use INT8 quantization.

• Performance issues

  • For Mixture-of-Experts (MoE) models, you can use the Autotune tool to improve performance. You can further optimize performance in vLLM by enabling a fused_moe kernel through a hardware-specific configuration file.

• Quantization issues

  • By default, vLLM uses the Marlin kernel to accelerate inference for GPTQ, AWQ, W8A8, and WOQ quantization. You can explicitly specify a quantization method, such as gptq_acext or awq_acext, to use other vLLM kernels. However, the acext library does not currently support quantization with act_order=True or GPTQ-Int8 weights. This limitation will be addressed in a future release.

  • vLLM uses vllm-flash-attn, xformers, or flashinfer as its attention backend. The default backend, vllm-flash-attn, is currently slower than the native GPU implementation. This is planned for future optimization.

  • vLLM implements A8W8 quantization by integrating with the acext library, and its performance requires further optimization.

  • Currently, performance with FP8 quantization is slow because it has not been optimized for PG1. The support timeline is to be determined.

• Stability issues

  • vLLM may randomly exit after a multi-GPU inference task is complete. This is a known upstream community issue. For details, see the open issue.