Deploy a DeepSeek distilled model inference service by using ACK Edge and virtual nodes-Container Service for Kubernetes(ACK)-阿里云帮助中心

DeepSeek inference services require increasingly powerful GPUs. To address this, you can use an ACK Edge cluster to manage GPU machines in your on-premises data center (IDC). You can also use the cluster's virtual nodes to quickly access ACS Serverless GPU computing power in the cloud. This solution runs inference jobs on your on-premises GPUs first. If your on-premises GPU resources are insufficient, jobs are automatically scheduled to ACS Serverless GPUs in the cloud. This helps you meet your business growth needs while reducing costs.

Background

DeepSeek-R1

DeepSeek-R1 is DeepSeek's first-generation reasoning model, trained with large-scale reinforcement learning to improve LLM reasoning performance. It outperforms other closed-source models in mathematical reasoning and programming competitions, and its performance approaches or surpasses the OpenAI o1 series on certain tasks. DeepSeek-R1 also delivers strong results across knowledge-intensive tasks such as writing and Q&A.

DeepSeek also distills its reasoning capabilities into smaller models based on the Qwen and Llama architectures. The 14B distilled model outperforms the open-source QwQ-32B model, while the 32B and 70B distilled models set new benchmarks. For more information, see the DeepSeek AI GitHub repository.

vLLM

vLLM is a high-performance LLM inference framework that supports most widely used LLMs, including the Qwen series. It uses PagedAttention, continuous batching, and model quantization to maximize inference throughput. For more information, see the vLLM GitHub repository.

Arena

Arena is a lightweight client for managing Kubernetes-based machine learning tasks. It covers the full ML lifecycle—data preparation, model development, training, and prediction—and improves data scientists' productivity. Arena deeply integrates with Alibaba Cloud infrastructure services, including GPU sharing and Cloud Parallel File Storage (CPFS), and can run Alibaba Cloud-optimized deep learning frameworks to maximize the performance and cost benefits of Alibaba Cloud heterogeneous devices. For more information, see the Arena GitHub repository.

Solution

Architecture

This solution uses the cloud-edge integrated management capabilities of an ACK Edge cluster. The Kubernetes control plane is hosted in the cloud, and machines in the on-premises data center (IDC) serve as data plane nodes for the Kubernetes cluster. This lets you manage your IDC machines through Kubernetes containerization. You can use the cluster's virtual nodes to quickly access ACS GPU computing power in the cloud. This approach unifies the management of on-premises and cloud compute resources and enables the dynamic allocation of computing jobs.

Connect your on-premises data center (IDC) resources to a cloud VPC through a leased line.
Add your on-premises IDC resources as edge nodes to an ACK Edge cluster. This lets you centrally manage and schedule your IDC services from the cloud.
Configure a custom ResourcePolicy for your services. This policy prioritizes scheduling jobs to on-premises IDC resources. If on-premises resources are insufficient, jobs are then scheduled to virtual nodes in the cloud.
Configure a Horizontal Pod Autoscaler (HPA) for your services. This automatically triggers a scale-out when resource usage reaches a specified threshold.

Benefits

High elasticity: Provides large-scale elastic scaling in seconds to quickly handle traffic peaks.
Fine-grained cost control: Pay as you go without needing to purchase your own servers. Costs are transparent and controllable.
Diverse elastic resources: Supports different instance types, such as CPU and GPU.

Prerequisites

Select a region to act as the central region and create an ACK Edge cluster in that region.
Install the ack-virtual-node component.
Install the ack-kserve component.
Configure the Arena client.
Deploy monitoring components and configure GPU monitoring metrics.
Create an edge node pool for a dedicated network and add your IDC resources to the edge node pool.

Procedure

Step 1: Prepare the model files

Note

Downloading and uploading the model typically takes 1 to 2 hours. To save time, submit a ticket to have the model files copied directly to your OSS bucket.

Download the DeepSeek-R1-Distill-Qwen-7B model from ModelScope using git-lfs.
Note
Check whether git-lfs is installed. If not, install it by running yum install git-lfs or apt-get install git-lfs. See Install Git Large File Storage.
```
git lfs install
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B.git
cd DeepSeek-R1-Distill-Qwen-7B/
git lfs pull
```

Create an OSS directory and upload the model files.

Note

To install ossutil, see Install ossutil.

ossutil mkdir oss://<your-bucket-name>/models/DeepSeek-R1-Distill-Qwen-7B
ossutil cp -r ./DeepSeek-R1-Distill-Qwen-7B oss://<your-bucket-name>/models/DeepSeek-R1-Distill-Qwen-7B

Create a PV and a PVC. Configure a persistent volume (PV) named llm-modeland a persistent volume claim (PVC) for the target cluster. For a full walkthrough, see Use an OSS static volume.

Use the following parameters when creating the PV:

Parameter	Value
PV type	OSS
Volume name	`llm-model`
Access certificate	The AccessKey ID and AccessKey secret for your OSS bucket
Bucket ID	The OSS bucket you created in the previous step
OSS path	`/models/DeepSeek-R1-Distill-Qwen-7B`

Use the following parameters when creating the PVC:

Parameter	Value
PVC type	OSS
Name	`llm-model`
Allocation mode	Existing volumes
Existing volumes	Select the PV created above

Yaml example:

apiVersion: v1
kind: Secret
metadata:
  name: oss-secret
stringData:
  akId: <your-oss-ak>       # AccessKey ID for the OSS bucket
  akSecret: <your-oss-sk>   # AccessKey secret for the OSS bucket
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: llm-model
  labels:
    alicloud-pvname: llm-model
spec:
  capacity:
    storage: 30Gi
  accessModes:
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: ossplugin.csi.alibabacloud.com
    volumeHandle: llm-model
    nodePublishSecretRef:
      name: oss-secret
      namespace: default
    volumeAttributes:
      bucket: <your-bucket-name>       # Name of the OSS bucket
      url: <your-bucket-endpoint>      # Endpoint, for example, oss-cn-hangzhou-internal.aliyuncs.com
      otherOpts: "-o umask=022 -o max_stat_cache_size=0 -o allow_other"
      path: <your-model-path>          # Model path, for example, /models/DeepSeek-R1-Distill-Qwen-7B/
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: llm-model
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 30Gi
  selector:
    matchLabels:
      alicloud-pvname: llm-model

Step 2: Create a custom scheduling policy ResourcePolicy

Create a ResourcePolicy custom resource definition (CRD) to define scheduling rules for elastic resource priority. In this example, the `labelSelector` matches the `isvc.deepseek-predictor` application to define a rule. This rule specifies that the application is preferentially scheduled to the edge IDC resource pool. If the edge IDC resources are insufficient, the application is scheduled to virtual nodes in the cloud. For more information about how to use ResourcePolicy, see Customize elastic resource scheduling priority.

Important

When you create the application pod later, you must add a label that matches the `labelSelector` in the policy to associate the pod with this scheduling policy.

Create a ResourcePolicy CRD and save it as a file named nginx-resourcepolicy.yaml.

apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
  name: deepseek
  namespace: default
spec:
  selector:
    app: isvc.deepseek-predictor # This must be associated with the label of the pod that you create later.
  strategy: prefer
  units:
  - resource: ecs
    nodeSelector:
      alibabacloud.com/nodepool-id: np*********  # The ID of the edge node pool.
  - resource: eci

Deploy the custom scheduling policy in the cluster to define the scheduling priority.
```
kubectl create -f nginx-resourcepolicy.yaml
```

Step 3: Deploy the model

Query the status of the nodes in the cluster.

kubectl get nodes -owide

Expected output:

NAME                            STATUS   ROLES    AGE     VERSION            INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                              KERNEL-VERSION           CONTAINER-RUNTIME
cn-hangzhou.10.4.XX.25           Ready    <none>   10d     v1.30.7-aliyun.1   10.4.0.25     <none>        Alibaba Cloud Linux 3.2104 U11 (OpenAnolis Edition)   5.10.134-18.al8.x86_64   containerd://1.6.36
cn-hangzhou.10.4.XX.26           Ready    <none>   10d     v1.30.7-aliyun.1   10.4.0.26     <none>        Alibaba Cloud Linux 3.2104 U11 (OpenAnolis Edition)   5.10.134-18.al8.x86_64   containerd://1.6.36
idc001                           Ready    <none>   31s     v1.30.7-aliyun.1   10.4.0.185    <none>        Alibaba Cloud Linux 3.2104 U11 (OpenAnolis Edition)   5.10.134-18.al8.x86_64   containerd://1.6.36
virtual-kubelet-cn-hangzhou-b    Ready    agent    7d21h   v1.30.7-aliyun.1   10.4.0.180    <none>        <unknown>                                             <unknown>                <unknown>

The expected output shows that the cluster has one IDC node (idc001) and one virtual node (virtual-kubelet-cn-hangzhou-b). The IDC node has one V100 GPU card.

Deploy the DeepSeek model inference service using the vLLM model inference framework.

arena serve kserve \
    --name=deepseek \
    --annotation=k8s.aliyun.com/eci-use-specs=ecs.gn6e-c12g1.3xlarge \
    --annotation=k8s.aliyun.com/eci-vswitch=vsw-*********,vsw-********* \
    --image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm:v0.6.6 \
    --gpus=1 \
    --cpu=4 \
    --memory=12Gi \
    --scale-metric=DCGM_CUSTOM_PROCESS_SM_UTIL \
    --scale-target=50 \
    --min-replicas=1  \
    --max-replicas=3  \
    --data=llm-model:/model/DeepSeek-R1-Distill-Qwen-7B \
    "vllm serve /model/DeepSeek-R1-Distill-Qwen-7B --port 8080 --trust-remote-code --served-model-name deepseek-r1 --max-model-len 32768 --gpu-memory-utilization 0.95 --enforce-eager --dtype=half"

The key parameters are described in the following table.

Parameter	Description	Example
`--name`	The name of the inference service to submit. The name must be globally unique.	deepseek
`--image`	The registry address of the inference service. This example uses the vLLM inference framework.	kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm:v0.6.6
`--gpus`	The number of GPU cards required by the inference service. The default value is 0.	1
`--cpu`	The number of CPUs required by the inference service.	4
`--memory`	The amount of memory required by the inference service.	12Gi
`--scale-metric`	The metric for application auto scaling. This example uses the GPU utilization metric `DCGM_CUSTOM_PROCESS_SM_UTIL` for application scaling. For more metrics, see 2. Configure HPA.	DCGM_CUSTOM_PROCESS_SM_UTIL
`--scale-target`	The target value for application scaling. When GPU utilization exceeds 50%, replica scale-out begins.	50
`--min-replicas`	The minimum number of replicas.	1
`--max-replicas`	The maximum number of replicas.	3
`--data`	Specifies the model address for the service. In this example, the model is stored in `llm-model` and mounted to the `/mnt/models/` directory in the container.	llm-model:/model/DeepSeek-R1-Distill-Qwen-7B \ "vllm serve /model/DeepSeek-R1-Distill-Qwen-7B --port 8080 --trust-remote-code --served-model-name deepseek-r1 --max-model-len 32768 --gpu-memory-utilization 0.95 --enforce-eager --dtype=half"

Expected output:

WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/bingchang/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/bingchang/.kube/config
horizontalpodautoscaler.autoscaling/deepseek-hpa created
inferenceservice.serving.kserve.io/deepseek created
INFO[0002] The Job deepseek has been submitted successfully
INFO[0002] You can run `arena serve get deepseek --type kserve -n default` to check the job status

View the details of the inference service.

arena serve get deepseek

Expected output:

Name:       deepseek
Namespace:  default
Type:       KServe
Version:    1
Desired:    1
Available:  1
Age:        1m
Address:    http://deepseek-default.example.com
Port:       :80
GPU:        1


Instances:
  NAME                                 STATUS   AGE  READY  RESTARTS  GPU  NODE
  ----                                 ------   ---  -----  --------  ---  ----
  deepseek-predictor-6b9455f8c5-wl5lc  Running  1m   1/1    0         1    idc001

The output shows that the application pod of the inference service is scheduled to an IDC node, which is consistent with the custom scheduling priority.

Send the following request to the service to verify that the inference service is running correctly. You can obtain the request address from the details of the Ingress resource automatically created by KServe.

curl -H "Host: deepseek-default.example.com" -H "Content-Type: application/json" http://<idc-node-ip>:<ingress-svc-nodeport>/v1/chat/completions -d '{"model": "deepseek-r1", "messages": [{"role": "user", "content": "Say this is a test!"}], "max_tokens": 512, "temperature": 0.7, "top_p": 0.9, "seed": 10}'

Expected output:

{"id":"chatcmpl-efc1225ad2f33cc39a8ddbc4039a41b9","object":"chat.completion","created":1739861087,"model":"deepseek-r1","choices":[{"index":0,"message":{"role":"assistant","content":"Okay, so I need to figure out how to say \"This is a test!\" in Spanish. Hmm, I'm not super fluent in Spanish, but I know some basic phrases. Let me think about how to approach this.\n\nFirst, I remember that \"test\" is \"prueba\" in Spanish. So maybe I can start with \"Esto es una prueba.\" But I'm not sure if that's the best way to say it. Maybe there's a more common expression or a different structure.\n\nWait, I think there's a phrase that's commonly used in tests. Isn't it something like \"This is a test.\" or \"This is a quiz.\"? I think the Spanish equivalent would be \"Este es un test.\" That sounds more natural. Let me check if that makes sense.\n\nI can also think about how people use phrases in tests. Maybe they use \"This is the test\" or \"This is an exam.\" So perhaps \"Este es el test.\" or \"Este es el examen.\" I'm not sure which one is more appropriate.\n\nI should also consider the grammar. \"This is a test\" is a simple statement, so the subject is \"this\" (using \"este\"), the verb is \"is\" (using \"es\"), and the object is \"a test\" (using \"un test\"). So putting it together, it would be \"Este es un test.\"\n\nWait, but sometimes people use \"This is the test\" when referring to an important one, so maybe \"Este es el test.\" But I'm not entirely sure if that's the correct structure. Let me think about other similar phrases.\n\nI also recall that in some contexts, people might say \"This is a practice test\" or \"This is a sample test.\" But since the user just said \"This is a test,\" the most straightforward translation would be \"Este es un test.\"\n\nI should also consider if there are any idiomatic expressions or common phrases that are used in this context. For example, \"This is the test\" is often used to mean a significant exam or evaluation, so \"Este es el test\" might be more appropriate in that context.\n\nBut I'm a bit confused because I'm not 100% sure about the correct structure. Maybe I should look up some examples. Oh, wait, I can't look things up right now, so I'll have to rely on my memory.\n\nI think the basic structure is subject + verb + object. So \"this\" (this is \"este","tool_calls":[]},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":11,"total_tokens":523,"completion_tokens":512,"prompt_tokens_details":null},"prompt_logprobs":null}

Step 4: Simulate peak business requests to trigger cloud elasticity

Use the stress testing tool Hey to send many requests to the deployed inference service.

hey -z 5m -c 5 \
-m POST -host deepseek-default.example.com \
-H "Content-Type: application/json" \
-d '{"model": "deepseek-r1", "messages": [{"role": "user", "content": "Say this is a test!"}], "max_tokens": 512, "temperature": 0.7, "top_p": 0.9, "seed": 10}' \
http://<idc-node-ip>:<ingress-svc-nodeport>/v1/chat/completions

The preceding request is sent to the existing pod. The high volume of requests triggers a pod scale-out when the GPU utilization exceeds the 50% threshold.

View the details of the inference service.

arena serve get deepseek

Expected output:

Name:       deepseek
Namespace:  default
Type:       KServe
Version:    1
Desired:    3
Available:  2
Age:        18m
Address:    http://deepseek-default.example.com
Port:       :80
GPU:        3


Instances:
  NAME                                 STATUS   AGE  READY  RESTARTS  GPU  NODE
  ----                                 ------   ---  -----  --------  ---  ----
  deepseek-predictor-6b9455f8c5-dtzdv  Running  1m   0/1    0         1    virtual-kubelet-cn-hangzhou-h
  deepseek-predictor-6b9455f8c5-wl5lc  Running  18m  1/1    0         1    idc001
  deepseek-predictor-6b9455f8c5-zmpg8  Running  5m   1/1    0         1    virtual-kubelet-cn-hangzhou-h

At this point, two pod replicas for the inference job have been scaled out to the virtual node.