Build a QwQ-32B model inference service using ACS GPU computing power-Container Compute Service-阿里云帮助中心

When using Container Compute Service (ACS) for GPU computing, you get an out-of-the-box experience without needing to understand the underlying hardware or manage and configure GPU nodes. ACS is simple to deploy and supports pay-as-you-go billing, making it ideal for large language model (LLM) inference tasks and helping reduce inference costs. This topic describes how to use ACS GPU computing power to deploy a production-ready QwQ-32B model inference service and expose it through the Open WebUI interface.

Background

QwQ-32B model

Alibaba Cloud's newly released QwQ-32B model significantly improves inference capabilities through reinforcement learning. With 32 billion parameters, its performance rivals that of the full-capacity DeepSeek-R1 model with 671 billion parameters. On core benchmarks such as AIME 24/25 and livecodebench, along with general-purpose metrics like IFEval and LiveBench, QwQ-32B matches the full-capacity DeepSeek-R1. All metrics substantially exceed those of DeepSeek-R1-Distill-Qwen-32B, which is also based on Qwen2.5-32B. For more information about the model, see QwQ-32B.

vLLM

vLLM is an efficient and easy-to-use framework for LLM inference services. vLLM supports various common large language models, including Qwen. vLLM achieves high efficiency in LLM inference through optimization techniques such as PagedAttention, continuous batching, and model quantization. For more information about the vLLM framework, see the vLLM GitHub repository.

Open WebUI

Open WebUI is an extensible, feature-rich, and user-friendly self-hosted AI platform designed to run entirely offline. It supports various LLM runtimes, such as Ollama and OpenAI-compatible APIs, and includes a built-in inference engine for retrieval-augmented generation (RAG), making it a powerful AI deployment solution.

Prerequisites

If this is your first time using Container Compute Service (ACS), you must grant the default system role to the service account. This role is required for ACS to call related services, such as ECS, OSS, NAS, CPFS, and SLB, create clusters, and save logs. For more information, see Use Container Compute Service for the first time.
You have connected to a Kubernetes cluster using kubectl.

GPU instance types and cost estimation

During inference, model parameters primarily consume GPU memory. Use the following formula to estimate memory requirements:

Model parameter count: 32B (32 billion). Default precision uses 16-bit floating-point numbers, which equals 2 bytes per parameter (16 bits ÷ 8 bits per byte = 2 bytes).

$Base GPU memory = 32 * 1 0^{9} * 2 bytes \approx 59.6 G i B$

In addition to memory used by the loaded model, account for KV cache size during computation and GPU utilization. Typically, reserve extra buffer space. We recommend using resources with at least 80 GiB of GPU memory: 1 GPU, 16 vCPUs, and 128 GiB memory. Refer to the Recommended instance types and GPU compute instance specifications to select an appropriate instance type. For details on calculating costs for ACS GPU instances, see Billing overview.

Note

ACS GPU instances follow the same ACS pod specification alignment logic.
ACS pods include 30 GiB of free temporary storage space (EphemeralStorage) by default. The inference image used in this topic, inference-nv-pytorch:25.02-vllm0.7.2-sglang0.4.3.post2-pytorch2.5-cuda12.4-20250305-serverless, occupies approximately 9.8 GiB. If this space is insufficient, you can increase the temporary storage size. For details, see Increase temporary storage space.

Procedure

Note

You can submit a ticket to quickly obtain the model files and supported GPU models.

Model file: QwQ-32B. The model file is approximately 120 GB. Downloading and uploading typically takes 2–3 hours. Submitting a ticket lets Alibaba Cloud quickly copy the model file to your OSS bucket.
GPU model: Replace the label alibabacloud.com/gpu-model-series: <example-model> with a specific GPU model supported by ACS. For details, see Specify ACS GPU computing power.

Step 1: Prepare QwQ-32B model data

Large language models require significant disk space due to their massive parameter counts. We recommend using NAS or OSS persistent volumes to store model files. The following steps use OSS as an example.

Run the following commands to download the QwQ-32B model.

Note
Confirm that git-lfs is installed. If not, run yum install git-lfs or apt-get install git-lfs to install it. For other installation methods, see Install git-lfs.
```
git lfs install
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/Qwen/QwQ-32B.git
cd QwQ-32B
git lfs pull
```
Create a directory in OSS and upload the model.

Note
For information about installing and using ossutil, see Install ossutil.
```
ossutil mkdir oss://<your-bucket-name>/models/QwQ-32B
ossutil cp -r ./QwQ-32B oss://<your-bucket-name>/models/QwQ-32B
```

Create a persistent volume (PV) and persistent volume claim (PVC). Configure a PV named llm-model and a PVC for your destination cluster. For details, see Use OSS static persistent volumes.

Console example

The following table shows basic PV configuration settings:

Configuration item	Description
Volume type	OSS
Name	llm-model
Access credentials	Configure the AccessKey ID and AccessKey secret for accessing OSS.
Bucket ID	Select the OSS bucket created in the previous step.
OSS Path	Select the model path, such as `/models/QwQ-32B`.

The following table shows basic PVC configuration settings:

Configuration item	Description
Persistent Volume Claim Type	OSS
Name	llm-model
Allocation mode	Select an existing volume.
Existing volume	Click the link to select the PV you created.

kubectl example

The following YAML shows an example configuration:

apiVersion: v1
kind: Secret
metadata:
  name: oss-secret
stringData:
  akId: <your-oss-ak> # Configure the AccessKey ID for accessing OSS
  akSecret: <your-oss-sk> # Configure the AccessKey secret for accessing OSS
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: llm-model
  labels:
    alicloud-pvname: llm-model
spec:
  capacity:
    storage: 30Gi 
  accessModes:
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: ossplugin.csi.alibabacloud.com
    volumeHandle: llm-model
    nodePublishSecretRef:
      name: oss-secret
      namespace: default
    volumeAttributes:
      bucket: <your-bucket-name> # Bucket name
      url: <your-bucket-endpoint> # Endpoint, such as oss-cn-hangzhou-internal.aliyuncs.com
      otherOpts: "-o umask=022 -o max_stat_cache_size=0 -o allow_other"
      path: <your-model-path> # In this example, /models/QwQ-32B/
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: llm-model
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 30Gi
  selector:
    matchLabels:
      alicloud-pvname: llm-model

Step 2: Deploy the model

Run the following command to deploy the QwQ-32B inference service using the vLLM framework.

This service exposes an OpenAI-compatible HTTP API. The command mounts the model parameter files as a special dataset to a specific path (/model/QwQ-32B) in the inference container. The --max-model-len parameter sets the maximum token length the model can process. Increasing this value improves conversation quality but may consume more GPU memory.

Note

egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/{image:tag} is a public registry address. To reduce image pull time, we recommend using VPC to accelerate AI container image pulls.

kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: qwq-32b
    alibabacloud.com/compute-class: gpu
    alibabacloud.com/compute-qos: default
    alibabacloud.com/gpu-model-series: <example-model>
  name: qwq-32b
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: qwq-32b
  template:
    metadata:
      labels:
        app: qwq-32b
        alibabacloud.com/compute-class: gpu
        alibabacloud.com/compute-qos: default
        alibabacloud.com/gpu-model-series: <example-model>
    spec:
      volumes:
        - name: model
          persistentVolumeClaim:
            claimName: llm-model
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 30Gi
      containers:
      - command:
        - sh
        - -c
        - vllm serve /models/QwQ-32B --port 8000 --trust-remote-code --served-model-name qwq-32b --max-model-len 32768 --gpu-memory-utilization 0.95 --enforce-eager
        image: egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:25.02-vllm0.7.2-sglang0.4.3.post2-pytorch2.5-cuda12.4-20250305-serverless
        name: vllm
        ports:
        - containerPort: 8000
        readinessProbe:
          tcpSocket:
            port: 8000
          initialDelaySeconds: 30
          periodSeconds: 30
        resources:
          limits:
            nvidia.com/gpu: "1"
            cpu: "16"
            memory: 128G
        volumeMounts:
          - mountPath: /models/QwQ-32B
            name: model
          - mountPath: /dev/shm
            name: dshm
---
apiVersion: v1
kind: Service
metadata:
  name: qwq-32b-v1
spec:
  type: ClusterIP
  ports:
  - port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    app: qwq-32b
EOF

Step 3: Deploy Open WebUI

Run the following command to create the Open WebUI application and service.

kubectl apply -f- << EOF 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: openwebui
spec:
  replicas: 1
  selector:
    matchLabels:
      app: openwebui
  template:
    metadata:
      labels:
        app: openwebui
    spec:
      containers:
      - env:
        - name: ENABLE_OPENAI_API
          value: "True"
        - name: ENABLE_OLLAMA_API
          value: "False"
        - name: OPENAI_API_BASE_URL
          value: http://qwq-32b-v1:8000/v1
        - name: ENABLE_AUTOCOMPLETE_GENERATION
          value: "False"
        - name: ENABLE_TAGS_GENERATION
          value: "False"
        image: kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/open-webui:main
        name: openwebui
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - mountPath: /app/backend/data
          name: data-volume
      volumes:
      - emptyDir: {}
        name: data-volume
---
apiVersion: v1
kind: Service
metadata:
  name: openwebui
  labels:
    app: openwebui
spec:
  type: ClusterIP
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: openwebui
EOF

Step 4: Verify the inference service

Use kubectl port-forward to create a port forwarding tunnel between the inference service and your local environment.

Note
The port forwarding established by kubectl port-forward lacks production-grade reliability, security, and scalability. Use it only for development and debugging, not in production environments. For production networking solutions in Kubernetes clusters, see Ingress management.
```
kubectl port-forward svc/openwebui 8080:8080
```
Expected output:
```
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
```
Access http://localhost:8080 and log on to the Open WebUI page.

On first login, set up an administrator account and password. Enter a prompt. The expected result is as follows.

After entering a prompt, the qwq-32b model returns an inference result. The page displays the model's reasoning process and generated response, confirming successful deployment.

(Optional) Step 5: Stress test the inference service

Note

Downloading the stress testing dataset requires public network access. For details, see Enable public network access for clusters or Attach EIPs to pods using annotations.

Run the following command to create the stress testing tool.

kubectl apply -f- <<EOF 
apiVersion: apps/v1 
kind: Deployment
metadata:
  name: vllm-benchmark
  labels:
    app: vllm-benchmark
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-benchmark
  template:
    metadata:
      labels:
        app: vllm-benchmark
    spec:
      volumes:
      - name: llm-model
        persistentVolumeClaim:
          claimName: llm-model
      containers:
      - name: vllm-benchmark
        image: kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm-benchmark:v1
        command:
        - "sh"
        - "-c"
        - "sleep inf"
        volumeMounts:
        - mountPath: /models/QwQ-32B
          name: llm-model
EOF

Enter the stress testing pod and download the dataset.

# Run the following command to enter the benchmark pod
PODNAME=$(kubectl get po -o custom-columns=":metadata.name"|grep "vllm-benchmark")
kubectl exec -it $PODNAME -- bash
# Download the stress testing dataset
pip3 install modelscope
modelscope download --dataset gliang1001/ShareGPT_V3_unfiltered_cleaned_split ShareGPT_V3_unfiltered_cleaned_split.json --local_dir /root/

Run the stress test.

# Run the stress test with input_length=4096, tp=4, output_length=512, concurrency=8, num_prompts=80
python3 /root/vllm/benchmarks/benchmark_serving.py \
--backend vllm \
--model /models/QwQ-32B \
--served-model-name qwq-32b \
--trust-remote-code \
--dataset-name random \
--dataset-path /root/ShareGPT_V3_unfiltered_cleaned_split.json \
--random-input-len 4096 \
--random-output-len 512 \
--random-range-ratio 1 \
--num-prompts 80 \
--max-concurrency 8 \
--host qwq-32b-v1 \
--port 8000 \
--endpoint /v1/completions \
--save-result \
2>&1 | tee benchmark_serving.txt

Expected output:

Starting initial single prompt test run...
Initial test run completed. Starting main benchmark run...
Traffic request rate: inf
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: 8
100%|██████████| 80/80 [07:44<00:00,  5.81s/it]
============ Serving Benchmark Result ============
Successful requests:                     80        
Benchmark duration (s):                  464.74    
Total input tokens:                      327680    
Total generated tokens:                  39554     
Request throughput (req/s):              0.17      
Output token throughput (tok/s):         85.11     
Total Token throughput (tok/s):          790.18    
---------------Time to First Token----------------
Mean TTFT (ms):                          10315.97  
Median TTFT (ms):                        12470.54  
P99 TTFT (ms):                           17580.34  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          71.03     
Median TPOT (ms):                        66.24     
P99 TPOT (ms):                           95.95     
---------------Inter-token Latency----------------
Mean ITL (ms):                           71.02     
Median ITL (ms):                         58.12     
P99 ITL (ms):                            60.26     
==================================================

(Optional) Step 6: Clean up the environment

If you no longer need the inference service deployed in this topic, clean up the environment promptly.

Delete the inference workloads and services.

kubectl delete deployment qwq-32b
kubectl delete service qwq-32b-v1
kubectl delete deployment openwebui
kubectl delete service openwebui
kubectl delete deployment vllm-benchmark

Delete the PV and PVC.

kubectl delete pvc llm-model
kubectl delete pv llm-model

Expected output:

persistentvolumeclaim "llm-model" deleted
persistentvolume "llm-model" deleted

References

Alibaba Cloud Container Compute Service (ACS) is integrated into Container Service for Kubernetes. You can quickly use ACS GPU container computing power with ACK managed cluster Pro.
The following topics describe building DeepSeek model inference services using ACS GPU computing power:
ACS AI container images are specialized images for ACS clusters using GPU instances. For information about currently available images, see ACS AI container image release notes.