Quickly deploy a DeepSeek V3/R1 inference service on a PPU using ACS-Container Compute Service(ACS)-阿里云帮助中心

This topic explains how to use Container Compute Service (ACS) to quickly deploy DeepSeek V3/R1 inference services on PPUs.

Prerequisites

You have signed up for an Alibaba Cloud account and completed individual real-name verification.
You have completed the initial setup for ACS, which involves activating the service and granting it the required permissions to access cloud resources.
You have activated OSS. In this guide, an OSS storage volume is used to persist model files.

Create an ACS cluster

This section describes how to quickly create an ACS cluster by configuring the main parameters.

For a detailed description of all configuration parameters for creating an ACS cluster, see Create an ACS cluster.

Log on to the ACS console. In the left navigation pane, click Clusters.
On the Clusters page, click Create Kubernetes Cluster in the upper-left corner.
On the Create Cluster page, configure the following settings. You can leave the other settings at their default values.
Parameter
Description
Example value
Cluster name
Enter a name for the cluster.
ACS-PPU-Inference
Region
Select the region where the cluster is located.
China (Ulanqab)
Click Confirm Configuration, and once all dependency checks pass, click Create Cluster.
Creating the cluster takes about 5 to 10 minutes.

Create a storage volume

Due to their large number of parameters, large language models require significant disk space to store their model files. We recommend that you create a NAS or OSS storage volume to store these files. For more information, see Use ACS to quickly create a data storage volume for large language models.

Download the DeepSeek-V3-bf16 model files: https://hf-mirror.com/opensourcerelease/DeepSeek-V3-bf16/tree/main
Download the DeepSeek-R1-bf16 model files:https://hf-mirror.com/opensourcerelease/DeepSeek-R1-bf16/tree/main
Download the DeepSeek-V3 FP8 model files: https://hf-mirror.com/deepseek-ai/DeepSeek-V3/tree/main
Download the DeepSeek-R1 FP8 model files:https://hf-mirror.com/deepseek-ai/DeepSeek-R1/tree/main

Note

PPU supports native DeepSeek V3/R1 FP8 models and INT8 W8A8 quantized models starting from the inference-xpu-pytorch 25.05 vLLM inference image (vLLM 0.8.5).
Starting from the inference-xpu-pytorch 25.08 version (vLLM 0.9.1), PPU supports the DeepSeek-V3.1 model. The vLLM server startup command is the same as the command for DeepSeek-R1 shown below.

Deploy inference service

Use the kubectl command-line tool or the ACS console to create a Deployment that uses PPU compute resources to run the large language model inference workload. To reduce image pull time, use a VPC to accelerate AI container image pulls.

On the Cluster List page in the ACS console, click the cluster name ACS-PPU-Inference. In the left-side navigation pane, choose Workloads > Stateless, and then click Create from YAML.

This section uses an OSS storage volume as an example to deploy services for the DeepSeek-R1-bf16, native DeepSeek-R1 FP8, and DeepSeek-R1 INT8 quantized models. Enter the YAML content for your desired model and click Create.

Note

You only need to add the --no-enable-prefix-caching parameter when starting the vLLM server with the inference-xpu-pytorch 25.05 vLLM inference image (vLLM 0.8.5). Adjust other parameters to suit your needs.
To reduce image pull time, use a VPC to accelerate AI container image pulls.

DeepSeek-R1-bf16

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: llm-test
  name: llm-test
  namespace: default
spec:
  progressDeadlineSeconds: 6000
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: llm-test
  template:
    metadata:
      labels:
        alibabacloud.com/compute-class: gpu
        alibabacloud.com/gpu-model-series: PPU810E
        alibabacloud.com/compute-qos: default
        app: llm-test
    spec:
      imagePullSecrets:
      - name: acs-image-secret # Must match the name of your image pull Secret.
      containers:
      - command:
        - sh
        - -c
        - vllm serve /mnt/llms_data/DeepSeek-R1-bf16/ --tensor-parallel-size 16 --gpu-memory-utilization 0.98 --max-model-len 5500 --trust-remote-code --no-enable-prefix-caching
 # /mnt/llms_data/DeepSeek-R1-bf16/ is the model path in the Pod.
        image: egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.05-v1.5.1-vllm0.8.5-torch2.6-cu126-20250604
        imagePullPolicy: IfNotPresent
        name: llm-test
        resources:
          limits:
            cpu: 176
            memory: 1800G
            alibabacloud.com/ppu: 16
            ephemeral-storage: 200Gi
          requests:
            cpu: 176
            memory: 1800G
            alibabacloud.com/ppu: 16
            ephemeral-storage: 200Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
          - mountPath: /mnt # The mount path for OSS.
            name: data
          - mountPath: /dev/shm
            name: cache-volume
          - mountPath: /ppu-data
            name: ephemeral
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: deepseek-oss # 'deepseek-oss' is the name of the persistent volume claim (PVC) that uses OSS.
        - name: cache-volume
          emptyDir:
            medium: Memory
            sizeLimit: 500G
        - name: ephemeral
          emptyDir:
            sizeLimit: 200G
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "internet"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: ipv4
  labels:
    app: llm-test
  name: svc-llm
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: serving
    port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    app: llm-test
  type: LoadBalancer

Native DeepSeek-R1 FP8

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: llm-test
  name: llm-test
  namespace: default
spec:
  progressDeadlineSeconds: 6000
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: llm-test
  template:
    metadata:
      labels:
        alibabacloud.com/compute-class: gpu
        alibabacloud.com/gpu-model-series: PPU810E
        alibabacloud.com/compute-qos: default
        app: llm-test
    spec:
      imagePullSecrets:
      - name: acs-image-secret # Must match the name of your image pull Secret.
      containers:
      - command:
        - sh
        - -c
        - vllm serve /mnt/llms_data/DeepSeek-R1/ --tensor-parallel-size 16 --gpu-memory-utilization 0.98 --trust-remote-code  --quantization moe_wna16 --no-enable-prefix-caching
        # /mnt/llms_data/DeepSeek-R1/ is the model path in the Pod.
        image: egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.05-v1.5.1-vllm0.8.5-torch2.6-cu126-20250604
        imagePullPolicy: IfNotPresent
        name: llm-test
        resources:
          limits:
            cpu: 176
            memory: 1800G
            alibabacloud.com/ppu: 16
            ephemeral-storage: 200Gi
          requests:
            cpu: 176
            memory: 1800G
            alibabacloud.com/ppu: 16
            ephemeral-storage: 200Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
          - mountPath: /mnt # The mount path for OSS.
            name: data
          - mountPath: /dev/shm
            name: cache-volume
          - mountPath: /ppu-data
            name: ephemeral
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: deepseek-oss # 'deepseek-oss' is the name of the persistent volume claim (PVC) that uses OSS.
        - name: cache-volume
          emptyDir:
            medium: Memory
            sizeLimit: 500G
        - name: ephemeral
          emptyDir:
            sizeLimit: 200G
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "internet"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: ipv4
  labels:
    app: llm-test
  name: svc-llm
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: serving
    port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    app: llm-test
  type: LoadBalancer

Important

If you use the inference-xpu-pytorch 25.06 vLLM inference image (vLLM 0.8.5), you must modify the config.json file of the DeepSeek-R1 FP8 model.

In the config.json file, find the quantization_config section:

"quantization_config": {
    "activation_scheme": "dynamic",
    "fmt": "e4m3",
    "quant_method": "fp8",
    "weight_block_size": [
      128,
      128
    ]
  },

Replace quantization_config with the following content:

"quantization_config": {
    "bits": 8,
    "group_size": 128,
    "desc_act": false,
    "quant_method": "gptq-fp8",
    "sym": true
  },

If you use the inference-xpu-pytorch 25.08 version, you do not need to modify the config.json file for the DeepSeek-R1 FP8 model, and the command does not require the --quantization moe_wna16 parameter.

DeepSeek-R1 INT8

Note

Starting from the PPU inference-xpu-pytorch 25.05 vLLM inference image (vLLM 0.8.5), DeepSeek V3/R1 models support the AWQ (w4a16), GPTQ (w4a16, w8a16), and per-token/per-channel a8w8 (int8) quantization schemes. To run models quantized with AWQ and GPTQ on vLLM 0.8.5, you must add the --quantization moe_wna16 parameter. The deployment procedure for W8A8-Channel-INT8 quantized models is the same as for non-quantized models.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: llm-test
  name: llm-test
  namespace: default
spec:
  progressDeadlineSeconds: 6000
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: llm-test
  template:
    metadata:
      labels:
        alibabacloud.com/compute-class: gpu
        alibabacloud.com/gpu-model-series: PPU810E
        alibabacloud.com/compute-qos: default
        app: llm-test
    spec:
      imagePullSecrets:
      - name: acs-image-secret # Must match the name of your image pull Secret.
      containers:
      - command:
        - sh
        - -c
        - vllm serve /mnt/llms_data/DeepSeek-R1-GPTQ-INT8/ --tensor-parallel-size 16 --gpu-memory-utilization 0.98 --trust-remote-code --no-enable-prefix-caching --quantization moe_wna16
 # /mnt/llms_data/DeepSeek-R1-GPTQ-INT8/ is the model path in the Pod.
        image: egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-xpu-pytorch:25.05-v1.5.1-vllm0.8.5-torch2.6-cu126-20250604
        imagePullPolicy: IfNotPresent
        name: llm-test
        resources:
          limits:
            cpu: 176
            memory: 1800G
            alibabacloud.com/ppu: 16
            ephemeral-storage: 200Gi
          requests:
            cpu: 176
            memory: 1800G
            alibabacloud.com/ppu: 16
            ephemeral-storage: 200Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
          - mountPath: /mnt # The mount path for OSS.
            name: data
          - mountPath: /dev/shm
            name: cache-volume
          - mountPath: /ppu-data
            name: ephemeral
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: deepseek-oss # 'deepseek-oss' is the name of the persistent volume claim (PVC) that uses OSS.
        - name: cache-volume
          emptyDir:
            medium: Memory
            sizeLimit: 500G
        - name: ephemeral
          emptyDir:
            sizeLimit: 200G
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "internet"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: ipv4
  labels:
    app: llm-test
  name: svc-llm
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: serving
    port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    app: llm-test
  type: LoadBalancer

Note

For information about GPU-related annotations, see ACS Pod instance overview.
For PPU specifications, see PPU Pod specifications.

The first deployment takes about 20 minutes to pull the image. You can check the Pod status on the Events tab by navigating to Workloads > Pods in your cluster.
The deployment is successful when the Pod status changes to Running. The vLLM serving service starts automatically after the Pod is created. You can check the logs on the Logs tab by navigating to Workloads > Pods in your cluster. The service has started successfully if you see output similar to the following.

Verify inference service

In the Container Compute Service console, navigate to Networking > Services. Find your svc-llm Service. The value in the External IP column is the public IP address of the inference service.

To test the vLLM chat completion feature, run the following command.

# Replace IP with the External IP of the Service from the previous step.
# The 'model' parameter must be the model path inside the Pod.

curl http://IP:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "/mnt/llms_data/DeepSeek-R1-bf16/",  
    "messages": [
    {"role": "system", "content": "You are a friendly AI assistant."},
    {"role": "user", "content": "Describe deep learning."}
    ]}'

Expected output:

Parameter	Description	Example value
Cluster name	Enter a name for the cluster.	`ACS-PPU-Inference`
Region	Select the region where the cluster is located.	`China (Ulanqab)`