Deploy a QwQ-32B inference service on ACS

Prepare a volume and model files

1. Large language models (LLMs) require significant disk space to store model files due to their large number of parameters. We recommend creating persistent storage for the model files by following the instructions in Use ACS to quickly create a data volume for a large language model. This example uses OSS, which by default creates an OSS bucket, a persistent volume (oss-pv), a persistent volume claim (oss-pvc), and a temporary pod to download the model files.

2. Connect to the temporary pod from your terminal and run the following commands to download the model files to the OSS-mounted local path, which is the /data directory in this example.

   pip install modelscope
   modelscope download --model Qwen/QwQ-32B --local_dir /data/QwQ-32B

Deploy the inference service

Follow these steps in the ACS console to create a deployment for the large model inference workload.

1. On the Clusters page, click the name of the target cluster. In the left navigation pane, choose Workloads > Custom Resources.

2. Click Create from YAML, paste the following YAML content, and then click Create.

   This example shows how to use a VPC to accelerate pulling AI container images to reduce the image pull time. Replace the address with the VPC image address in the same region based on your actual situation.

   PPU

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: llm-test
     name: llm-test
     namespace: default
   spec:
     progressDeadlineSeconds: 600
     replicas: 1
     revisionHistoryLimit: 10
     selector:
       matchLabels:
         app: llm-test
     template:
       metadata:
         labels:
           alibabacloud.com/compute-class: gpu
           alibabacloud.com/gpu-model-series: PPU810E
           alibabacloud.com/compute-qos: default
           app: llm-test
       spec:
         containers:
           - command:
             - sh
             - -c
             - python3 -m vllm.entrypoints.openai.api_server --model /mnt/QwQ-32B --tensor-parallel-size 1 --trust-remote-code --max-model-len 64000 --gpu-memory-utilization 0.95
             # /mnt/QwQ-32B is the model path in the pod.
             image: acs-registry-vpc.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.02-v1.4.2-vllm0.7.2-torch2.5-cuda12.6-20250304
             imagePullPolicy: IfNotPresent
             name: llm-test
             resources:
               limits:
                 cpu: 10
                 memory: 80G
                 alibabacloud.com/ppu: 1
                 ephemeral-storage: 200Gi
               requests:
                 cpu: 10
                 memory: 80G
                 alibabacloud.com/ppu: 1
                 ephemeral-storage: 200Gi
             terminationMessagePath: /dev/termination-log
             terminationMessagePolicy: File
             volumeMounts:
               - mountPath: /mnt # OSS mount path
                 name: data
               - mountPath: /ppu-data
                 name: ephemeral
         dnsPolicy: ClusterFirst
         restartPolicy: Always
         schedulerName: default-scheduler
         securityContext: {}
         terminationGracePeriodSeconds: 30
         volumes:
           - name: data
             persistentVolumeClaim:
               claimName: oss-pvc # The name of your persistent volume claim.
           - name: ephemeral
             emptyDir:
               sizeLimit: 200G
   ---
   apiVersion: v1
   kind: Service
   metadata:
     annotations:
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "internet"
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: ipv4
     labels:
       app: llm-test
     name: svc-llm
     namespace: default
   spec:
     externalTrafficPolicy: Local
     ports:
     - name: serving
       port: 8000
       protocol: TCP
       targetPort: 8000
     selector:
       app: llm-test
     type: LoadBalancer

   GU8TF

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: llm-test
     name: llm-test
     namespace: default
   spec:
     progressDeadlineSeconds: 600
     replicas: 1
     revisionHistoryLimit: 10
     selector:
       matchLabels:
         app: llm-test
     template:
       metadata:
         labels:
           alibabacloud.com/compute-class: gpu
           alibabacloud.com/gpu-model-series: GU8TF
           alibabacloud.com/compute-qos: default
           app: llm-test
       spec:
         containers:
           - command:
             - sh
             - -c
             - python3 -m vllm.entrypoints.openai.api_server --model /mnt/QwQ-32B --tensor-parallel-size 1 --trust-remote-code --max-model-len 64000 --gpu-memory-utilization 0.95
             # /mnt/QwQ-32B is the model path in the pod.
             image: egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:25.02-vllm0.7.2-sglang0.4.3.post2-pytorch2.5-cuda12.4-erdma-20250224
             imagePullPolicy: IfNotPresent
             name: llm-test
             resources:
               limits:
                 cpu: 10
                 memory: 80G
                 nvidia.com/gpu: 1
                 ephemeral-storage: 200Gi
               requests:
                 cpu: 10
                 memory: 80G
                 nvidia.com/gpu: 1
                 ephemeral-storage: 200Gi
             terminationMessagePath: /dev/termination-log
             terminationMessagePolicy: File
             volumeMounts:
               - mountPath: /mnt # OSS mount path
                 name: data
               - mountPath: /gu8tf-data
                 name: ephemeral
         dnsPolicy: ClusterFirst
         restartPolicy: Always
         schedulerName: default-scheduler
         securityContext: {}
         terminationGracePeriodSeconds: 30
         volumes:
           - name: data
             persistentVolumeClaim:
               claimName: oss-pvc # The name of your persistent volume claim.
           - name: ephemeral
             emptyDir:
               sizeLimit: 200G
   ---
   apiVersion: v1
   kind: Service
   metadata:
     annotations:
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "internet"
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: ipv4
     labels:
       app: llm-test
     name: svc-llm
     namespace: default
   spec:
     externalTrafficPolicy: Local
     ports:
     - name: serving
       port: 8000
       protocol: TCP
       targetPort: 8000
     selector:
       app: llm-test
     type: LoadBalancer

   Note

   • For more information about GPU-related annotations, see ACS pod instances.

   • For more information about PPU specifications, see PPU pod specifications.

   • For more information about GU8TF specifications, see GPU pod specifications.

3. The initial image pull may take about 20 minutes. You can monitor the pod's status on the Events tab by navigating to Workloads > Pods in your cluster.

   

4. The pod is created successfully when its status changes to Running. After the pod is running, the vLLM serving service starts automatically. You can check the logs on the Logs tab by navigating to Workloads > Pods. The following output indicates that the service has started successfully. Loading the QwQ-32B model takes about 30 minutes.

   

Test the inference service

1. In the ACS console, navigate to Network > Services. Find the svc-llm service you created and note its public IP address.

2. Test the vLLM chat completion feature from your client.

   # Replace <IP> with the public IP address of the service you created in the previous step.
   # The "model" parameter specifies the path to the model in the pod.
   
   curl http://<IP>:8000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "/mnt/QwQ-32B",
       "messages": [
         {
           "role": "user",
           "content": "Write a letter to my daughter from the year 2035. Tell her to study technology, become a master of technology, and drive technological and economic development. She is currently in third grade."
         }
       ],
       "max_tokens": 1024,
       "temperature": 0.7,
       "top_p": 0.9,
       "seed": 10
     }'

   Expected output: