DocsICP FilingConsole
官方文档
首页

Using LLM inference images on ACS clusters

更新时间：
复制 MD 格式
产品详情
我的收藏

This guide explains how to use ACS LLM inference container images to create workloads on an ACS cluster and quickly build an LLM inference environment.

Prerequisites

  • You have created an ACS cluster.

  • You have created a NAS storage volume that contains the model files.

    Procedure

    1. You have activated Alibaba Cloud NAS.

    2. Prepare a NAS file system. Large language models (LLMs) require significant disk space for their model files. We recommend creating a NAS storage volume to persistently store these files. For more information, see Use a dynamic NAS storage volume. This guide uses a statically mounted NAS storage volume as an example. A storage capacity of 20 GiB is recommended.

      Create a persistent volume claim (PVC) with the following configurations. For more information, see Create a PVC.

      Parameter

      Example

      Name

      nas-test

      Allocation mode

      Select Create by Using Mount Target.

      Total capacity

      20Gi

      Mount target domain name

      08cxxxxxxec-wxxxxxxcn-wulanchabu.nas.aliyuncs.com

    3. Mount the NAS file system to an ECS instance. For more information, see Mount an NFS file system with one click.

      Run the following commands to download the model to the NAS file system:

      cd /mnt
pip install modelscope
modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct
      Note

      • The Qwen2.5-7B-Instruct model used in this guide is approximately 15 GB. It takes about 20 minutes to download over a 100 Mbps connection. You can use a more lightweight model, such as Qwen2.5-3B or Qwen2.5-0.5B, based on your requirements.

      • Because modelscope requires Python version ≥ 3.10 to run properly, we recommend using Ubuntu 22.04 as the ECS instance image, which has a default Python version of 3.10. You can also use other public or custom images, as long as the Python version is ≥ 3.10.

ACS LLM inference container images

Use one of the following ACS LLM inference container images when you configure workloads on your ACS instances.

Note

Image name

Image tag

Public image address

VPC image address

Description

inference-xpu-pytorch

25.03-v1.4.3-hotfix-vllm0.7.3-torch2.5-cu123-20250331

egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.03-v1.4.3-hotfix-vllm0.7.3-torch2.5-cu123-20250331

  • Example for China (Beijing) region:

    acs-registry-vpc.cn-beijing.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.03-v1.4.3-hotfix-vllm0.7.3-torch2.5-cu123-20250331

  • Example for China (Ulanqab) region:

    acs-registry-vpc.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.03-v1.4.3-hotfix-vllm0.7.3-torch2.5-cu123-20250331

  • Upgrades the PPU SDK in the base image to v1.4.3-hotfix.

  • Upgrades vLLM to v0.7.3 to improve inference performance for DeepSeek-V3/R1.

  • Upgrades SGLang to v0.4.3.

  • Upgrades ACCL-P to v1.4.3-8 and EIC to 1.3.7.1-v2503, incorporating new features and bug fixes from these versions.

Note

  • This image supports the vLLM framework for building large language model inference services. For more information, see vLLM.

  • You can build your own images based on this image and store them in your ACR repository to accelerate image pulls. This may incur additional fees.

  • For a list of the core AI libraries built into the image, see the latest image component list.

Deploy an LLM inference workload

Step 1: Configure a Deployment

Use the kubectl command-line tool or the ACS console to create a Deployment for the LLM inference workload.

Note

  • When you create the workload for the first time, the system pulls the image, which takes approximately 20 minutes. You can check the pod status on the Workloads > Pods > Events page of the destination cluster. The Qwen2.5-7B-Instruct model takes about 5 to 10 minutes to load. The actual time may vary depending on your deployment environment.

  • You can also create a pod directly for testing. The following sample YAML can be used to quickly start a Zhenwu-810E pod.

    Sample YAML for an LLM inference pod

    apiVersion: v1
kind: Pod
metadata:
  labels:
    alibabacloud.com/hpn-type: rdma
    alibabacloud.com/compute-class: gpu-hpn
    app: llm-test
  name: llm-test
spec:
  containers:
    - command:
      - sh
      - -c
      - python3 -m vllm.entrypoints.openai.api_server --model /mnt/Qwen2.5-7B-Instruct --trust-remote-code --tensor-parallel-size 8 --disable-custom-all-reduce   
      image: acs-registry-vpc.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.03-v1.4.3-hotfix-vllm0.7.3-torch2.5-cu123-20250331
      imagePullPolicy: Always
      name: llm-test
      resources:
        limits:
          cpu: 176
          memory: 1800G
          alibabacloud.com/ppu: 16
        requests:
          cpu: 176
          memory: 1800G
          alibabacloud.com/ppu: 16
      securityContext:
        capabilities: {}
        privileged: false
      terminationMessagePath: /dev/termination-log
      volumeMounts:
        - mountPath: /mnt
          name: data  
        - mountPath: /dev/shm
          name: cache-volume
  volumes:
  - name: data 
    persistentVolumeClaim:
      claimName: nas-pvc-fs # nas-pvc-fs is the persistent volume claim created from NAS.
  - name: cache-volume
    emptyDir:
      medium: Memory
      sizeLimit: 500G
  dnsPolicy: ClusterFirst
  restartPolicy: Always

Console

The following steps show how to create a Deployment. The procedure is similar for other types of workloads.

  1. Log on to the ACS console. In the left navigation pane, click Clusters.

  2. On the Clusters page, click the name of the target cluster. In the left navigation pane, choose Workloads > Deployments.

  3. Click Create from YAML, use the following YAML to create a Deployment, and then click Create.

    apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: llm-test
  name: llm-test
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: llm-test
  template:
    metadata:
      labels:
        alibabacloud.com/hpn-type: rdma
        alibabacloud.com/compute-class: gpu-hpn
        app: llm-test
    spec:
      containers:
        - command:
          - sh
          - -c
          - python3 -m vllm.entrypoints.openai.api_server --model /mnt/Qwen2.5-7B-Instruct --trust-remote-code  --tensor-parallel-size 8  --disable-custom-all-reduce
          # --tensor-parallel-size 8 enables 8-GPU parallelism.
          # /mnt/Qwen2.5-7B-Instruct is the model path in the pod.
          image: acs-registry-vpc.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.03-v1.4.3-hotfix-vllm0.7.3-torch2.5-cu123-20250331
          imagePullPolicy: IfNotPresent
          name: llm-test
          resources:
            limits:
              cpu: 176
              memory: 1800G
              alibabacloud.com/ppu: 16
            requests:
              cpu: 176
              memory: 1800G
              alibabacloud.com/ppu: 16
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /mnt # NAS mount path
              name: data
            - mountPath: /dev/shm
              name: cache-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: nas-pvc-fs # nas-pvc-fs is the persistent volume claim created from NAS.
        - name: cache-volume
          emptyDir:
            medium: Memory
            sizeLimit: 500G

  4. Return to the Deployments page to view the new Deployment.

    The Deployment is created successfully when the pod status changes to Running. After the pod starts, the vLLM serving service starts automatically.

  5. On the workload's Details > Logs tab, view the service logs. If output similar to the following is displayed, the service has started successfully.

    image.png

kubectl

  1. Connect to the cluster by using kubectl. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

  2. Save the following sample YAML content to a file named llm-test.yaml.

    apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: llm-test
  name: llm-test
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: llm-test
  template:
    metadata:
      labels:
        alibabacloud.com/hpn-type: rdma
        alibabacloud.com/compute-class: gpu-hpn
        app: llm-test
    spec:
      containers:
        - command:
          - sh
          - -c
          - python3 -m vllm.entrypoints.openai.api_server --model /mnt/Qwen2.5-7B-Instruct --trust-remote-code  --tensor-parallel-size 8  --disable-custom-all-reduce
          # --tensor-parallel-size 8 enables 8-GPU parallelism.
          # /mnt/Qwen2.5-7B-Instruct is the model path in the pod.
          image: acs-registry-vpc.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.03-v1.4.3-hotfix-vllm0.7.3-torch2.5-cu123-20250331
          imagePullPolicy: IfNotPresent
          name: llm-test
          resources:
            limits:
              cpu: 176
              memory: 1800G
              alibabacloud.com/ppu: 16
            requests:
              cpu: 176
              memory: 1800G
              alibabacloud.com/ppu: 16
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /mnt # NAS mount path
              name: data
            - mountPath: /dev/shm
              name: cache-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: nas-pvc-fs # nas-pvc-fs is the persistent volume claim created from NAS.
        - name: cache-volume
          emptyDir:
            medium: Memory
            sizeLimit: 500G

  3. Run the following command to deploy the Deployment from the YAML file:

    kubectl --kubeconfig config apply -f llm-test.yaml

  4. After the Deployment is created, run the following command to check if the pod has started successfully:

    The pod has started successfully if its status is Running.
    kubectl --kubeconfig config get pods

  5. After the pod starts, run the kubectl logs command to check if the vLLM service has started successfully.

    kubectl --kubeconfig config logs llm-test-68xxxxx9r5tf

Step 2: Create a service

  1. Log on to the ACS console. In the left navigation pane, click Clusters.

  2. On the Clusters page, click the name of the target cluster. In the left navigation pane, choose Network > Services.

  3. Click Create. Configure the llm-svc service with the following settings, associate it with the workload from the previous step, and then click OK.

    1. Associated workload: app: llm-test

    2. Port mapping:

      1. Name: serving

      2. Service port: 8000

      3. Container port: 8000

      4. Protocol: TCP

  4. Return to the services list to view the llm-svc service.

Step 3: Test LLM inference

  1. The following example uses the Qwen2.5-7B-Instruct model to test the vLLM chat inference feature:

    # Replace IP with the external IP address of the service created in the previous step.
# The 'model' parameter specifies the model path in the pod.

curl http://IP:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "/mnt/Qwen2.5-7B-Instruct",  
    "messages": [
    {"role": "system", "content": "You are a friendly AI assistant."},
    {"role": "user", "content": "Tell me about deep learning."}
    ]}'

Clean up resources

Use the kubectl command-line tool or the ACS console to clean up the resources.

  • Log on to the Container Compute Service console. On the Deployments, StorageClasses, and Persistent Volume Claims pages, delete the corresponding resources.

  • Use the kubectl command-line tool to delete the Deployment, persistent volume claim (PVC), and storage class.

    # Deleting the Deployment automatically cleans up the pods.
kubectl --kubeconfig config delete deployment llm-test

# Delete the persistent volume claim.
kubectl --kubeconfig config delete pvc nas-pvc-fs

# Delete the storage class.
kubectl --kubeconfig config delete sc alicloud-nas-fs
Previous：InferenceNext：Training