inference-xpu-pytorch 25.03, inference-xpu-pytorch 25.03

Main features and bug fixes

Image contents

Image assets

Quick start

The following example shows how to pull the inference-xpu-pytorch image using Docker and test the inference service with the Qwen2.5-7B-Instruct model.

1. Pull the inference image.

   docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

2. Download the open-source model in ModelScope format.

   pip install modelscope
   cd /mnt
   modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

3. Run the following command to start and enter the container.

   docker run -d -t --network=host --privileged --init --ipc=host \
   --ulimit memlock=-1 --ulimit stack=67108864  \
   -v /mnt/:/mnt/ \
   egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:[tag]

4. Run an inference test to verify the conversational inference feature of vLLM.

   1. Start the server.

      python3 -m vllm.entrypoints.openai.api_server \
      --model /mnt/Qwen2.5-7B-Instruct \
      --trust-remote-code --disable-custom-all-reduce \
      --tensor-parallel-size 1

   2. Run a test on the client.

      curl http://localhost:8000/v1/chat/completions \
          -H "Content-Type: application/json" \
          -d '{
          "model": "/mnt/Qwen2.5-7B-Instruct",  
          "messages": [
          {"role": "system", "content": "You are a friendly AI assistant."},
          {"role": "user", "content": "Tell me about deep learning."}
          ]}'

      The following output is returned:

      

      For more information about using vLLM, see the vLLM documentation.

Known issues

• DeepSeek issues

  • When testing the DeepSeek model, the --max-model-len setting must be < 80k. Otherwise, an Out of Memory (OOM) error may occur during runtime.

  • SGLang does not support running INT8 quantized models.

  • vllm 0.7.3 requires you to set the --max-num-batched-tokens parameter to avoid the framework's default limit of 2048. For more information, see the open-source community issue [Usage]:Input prompt (2501 tokens) is too long and exceeds limit of 2048 #13370.

• Performance issues

  • You can use the Autotune tool to improve the performance of Mixture-of-Experts (MoE) models. vLLM can launch a hardware-specific fused_moe kernel through a configuration file to boost performance.

• Quantization issues

  • vLLM uses the Marlin Kernel by default to accelerate quantized inference for GPTQ, AWQ, W8A8, and WOQ. You can explicitly specify a quantization method to use vLLM's original quantized inference kernels, such as Marlin, W8A8, awq, or gptq. For example, you can specify gptq_acext or awq_acext to perform quantized inference. However, acext currently does not support quantization with act_order=True or GPTQ-Int8 quantized weights. This will be optimized in the next version.

  • vLLM uses vllm-flash-attn, xformers, or flashinfer as its attention backend. By default, the performance of the vllm-flash-attn backend is not yet on par with the native GPU version. This will be addressed in a future release.

  • The A8W8 quantization in vLLM is implemented through integration with the acext library. Its performance needs further optimization.

  • Currently, FP8 quantization is very slow and has not been adapted or optimized for PPU. The support timeline is to be determined.

• Stability issues

  • When you use vLLM for multi-card inference, an error related to mp_distributed_executor may occur. To avoid this issue, use VLLM_WORKER_MULTIPROC_METHOD=spawn. For more information, see the following GitHub issue: [Bug]: When tensor_parallel_size>1, RuntimeError: Cannot re-initialize CUDA in forked subprocess. #6152.

  • After multi-card inference with vLLM is complete, a known community issue may cause a process to exit randomly. For more details, see the open-source community issue [Bug]: When using tp for inference, an error occurs: Worker VllmWorkerProcess pid 3283517 died, exit code: -15. #6145.