llm-inference-xpu 24.12-ZHENWU PPU Cloud Service(ppu)-阿里云帮助中心

This topic describes the release notes for llm-inference-xpu 24.12.

Key features and bug fixes

Key features

Upgraded the PPU SDK in the base image to version 1.4.0 hotfix2.
Upgraded vLLM to version v0.6.3.post1.

Bug fixes

None.

Image assets

Note

We recommend using a VPC to accelerate image pulls and reduce pull times.

Public image

egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/llm-inference-xpu:1.4.0-hotfix2-vllm0.6.3.post1-torch2.4-cuda12.3-ubuntu22.04-202412

VPC images

acs-registry-vpc.{region-id}.cr.aliyuncs.com/egslingjun/{image:tag}
{region-id} is the ID of the region where your Container Service for Kubernetes (ACK) cluster is located, such as cn-beijing or cn-wulanchabu.
{image:tag} is the actual image name and tag.

Image components

Version 24.12	llm-inference-xpu
Use case	LLM inference
Framework	PyTorch
Requirements	PPU SDK v1.4.0
System components	Ubuntu 22.04 Python 3.10 Torch 2.4.0 CUDA 12.3 vllm 0.6.3.post1+cu123 transformers 4.46.2 vllm-flash-attn 2.6.2+cu123

Quick start

This example uses the Qwen2.5 7B model.

Pull the container

docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/llm-inference-xpu:[tag]

Download the model

Download an open-source model from ModelScope.

pip install modelscope
cd /mnt
modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

Start and enter the container

docker run  --rm -it --gpus all --privileged --ipc=host \
--ulimit memlock=-1 --ulimit stack=67108864  \
-v /mnt/:/mnt/ \
egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/llm-inference-xpu:[tag]

Run an inference test

Start the server

python3 -m vllm.entrypoints.openai.api_server \
--model /mnt/Qwen2.5-7B-Instruct \
--trust-remote-code \
--tensor-parallel-size 1

Test on the client

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "/mnt/Qwen2.5-7B-Instruct",  
    "messages": [
    {"role": "system", "content": "You are a friendly AI assistant."},
    {"role": "user", "content": "Tell me about deep learning."}
    ]}'

Sample output:

For more information about using vLLM, see the vLLM documentation.

To learn how to use XPU large model inference images on Alibaba Cloud Stack, see Use LLM large model inference images on an Alibaba Cloud Stack cluster.

Known issues

MOE Autotune improves performance for Mixture-of-Experts (MoE) models. vLLM provides this optimization by launching kernels from a hardware-specific fused_moe kernel configuration file.
Quantization issues
- By default, vLLM uses the Marlin kernel to accelerate quantized inference for GPTQ, AWQ, W8A8, and Weight-Only Quantization (WOQ). You can also specify a quantization method to use other native vLLM kernels. For example, you can use gptq_acext or awq_acext for quantized inference. However, acext currently does not support quantization with act_order=True or GPTQ-Int8 weights. This will be optimized in a future release.
- The implementation of the Marlin quantization kernel in vLLM is hardware-dependent and designed for A100 GPUs. On PPUs, the performance of this ported kernel does not yet match the original A100 GPU version. This will be optimized in a future release.
- vLLM uses vllm-flash-attn, xformers, or flashinfer as the attention backend. The performance of the default backend, vllm-flash-attn, on PPUs does not yet match its performance on GPUs. This will be optimized in a future release.
- vLLM implements A8W8 quantization using the acext library, but its performance requires further optimization.
- Inference with FP8 quantization is currently very slow as it has not been adapted or optimized for PPUs. A future support plan has not yet been determined.
Stability issues
- After a multi-GPU inference task completes, vLLM may randomly exit with an error. This is a known community issue. For more details, see issue #6145 in the vLLM open-source project: https://github.com/vllm-project/vllm/issues/6145.