llm-inference-xpu 24.11

Public image

• egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/llm-inference-xpu:vllm0.5.3.post1-pytorch2.5.1-cuda12.4-ubuntu22.04

VPC images

• acs-registry-vpc.{region-id}.cr.aliyuncs.com/egslingjun/{image:tag}

  {region-id} is the ID of the region where your Container Service for Kubernetes (ACK) cluster is located, such as cn-beijing or cn-wulanchabu.

  {image:tag} is the actual image name and tag.

Quick start

This example uses the Qwen2.5 7B model.

1. Pull the container

   docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/llm-inference-xpu:[tag]

2. Download the model

   Download an open-source model from ModelScope.

   pip install modelscope
   cd /mnt
   modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

3. Start and enter the container

   docker run  --rm -it --gpus all --privileged --ipc=host \
   --ulimit memlock=-1 --ulimit stack=67108864  \
   -v /mnt/:/mnt/ \
   egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/llm-inference-xpu:[tag]

4. Run an inference test

   1. Start the server

      python3 -m vllm.entrypoints.openai.api_server \
      --model /mnt/Qwen2.5-7B-Instruct \
      --trust-remote-code \
      --tensor-parallel-size 1

   2. Test on the client

      curl http://localhost:8000/v1/chat/completions \
          -H "Content-Type: application/json" \
          -d '{
          "model": "/mnt/Qwen2.5-7B-Instruct",  
          "messages": [
          {"role": "system", "content": "You are a friendly AI assistant."},
          {"role": "user", "content": "Tell me about deep learning."}
          ]}'

      Sample output:

      

   For more information about using vLLM, see the vLLM documentation.

To learn how to use XPU large model inference images on Alibaba Cloud Stack, see Use LLM large model inference images on an Alibaba Cloud Stack cluster.