Deploy a DeepSeek distilled model inference service by using ACK Edge and virtual nodes-Container Service for Kubernetes(ACK)-阿里云帮助中心

DeepSeek inference services require increasingly powerful GPUs. To address this, you can use an ACK Edge cluster to manage GPU machines in your on-premises data center (IDC). You can also use the cluster's virtual nodes to quickly access ACS Serverless GPU computing power in the cloud. This solution runs inference jobs on your on-premises GPUs first. If your on-premises GPU resources are insufficient, jobs are automatically scheduled to ACS Serverless GPUs in the cloud. This helps you meet your business growth needs while reducing costs.

Background

DeepSeek-R1 model

The DeepSeek-R1 model is the first-generation reasoning model from DeepSeek. It is designed to enhance the reasoning capabilities of large language models through large-scale reinforcement learning. Experiments show that DeepSeek-R1 performs well on various tasks, such as mathematical reasoning and programming competitions. It not only surpasses other closed-source models but also approaches or exceeds the performance of the OpenAI-O1 series models on certain tasks. DeepSeek-R1 also excels in knowledge-based tasks and a wide range of other tasks, including creative writing and general Q&A. DeepSeek has also distilled these reasoning capabilities into smaller models. This improves the reasoning ability of existing models, such as Qwen and Llama, through fine-tuning. The distilled 14B model significantly outperforms the existing open-source QwQ-32B model, while the distilled 32B and 70B models have set new performance records. For more information about DeepSeek models, see the DeepSeek AI GitHub repository.

vLLM

vLLM is an efficient and easy-to-use framework for LLM inference services. vLLM supports various common large language models, including Qwen. vLLM achieves high efficiency in LLM inference through optimization techniques such as PagedAttention, continuous batching, and model quantization. For more information about the vLLM framework, see the vLLM GitHub repository.

Arena

Arena is a lightweight machine learning solution based on Kubernetes. It supports the entire machine learning lifecycle, including data preparation, model development, model training, and model prediction, to improve the efficiency of data scientists. Arena is deeply integrated with Alibaba Cloud's foundational cloud services and supports features such as GPU sharing and CPFS. It can run deep learning frameworks optimized by Alibaba Cloud to maximize the performance and cost-effectiveness of Alibaba Cloud's heterogeneous devices. For more information about Arena, see the Arena GitHub repository.

Solution

Architecture

This solution uses the cloud-edge integrated management capabilities of an ACK Edge cluster. The Kubernetes control plane is hosted in the cloud, and machines in the on-premises data center (IDC) serve as data plane nodes for the Kubernetes cluster. This lets you manage your IDC machines through Kubernetes containerization. You can use the cluster's virtual nodes to quickly access ACS GPU computing power in the cloud. This approach unifies the management of on-premises and cloud compute resources and enables the dynamic allocation of computing jobs.

Connect your on-premises data center (IDC) resources to a cloud VPC through a leased line.
Add your on-premises IDC resources as edge nodes to an ACK Edge cluster. This lets you centrally manage and schedule your IDC services from the cloud.
Configure a custom ResourcePolicy for your services. This policy prioritizes scheduling jobs to on-premises IDC resources. If on-premises resources are insufficient, jobs are then scheduled to virtual nodes in the cloud.
Configure a Horizontal Pod Autoscaler (HPA) for your services. This automatically triggers a scale-out when resource usage reaches a specified threshold.

Benefits

High elasticity: Provides large-scale elastic scaling in seconds to quickly handle traffic peaks.
Fine-grained cost control: Pay as you go without needing to purchase your own servers. Costs are transparent and controllable.
Diverse elastic resources: Supports different instance types, such as CPU and GPU.

Prerequisites

Select a region to act as the central region and create an ACK Edge cluster in that region.
Install the ack-virtual-node component.
Install the ack-kserve component.
Configure the Arena client.
Deploy monitoring components and configure GPU monitoring metrics.
Create an edge node pool for a dedicated network and add your IDC resources to the edge node pool.

Procedure

Step 1: Prepare the DeepSeek-R1-Distill-Qwen-7B model files

Note

Downloading and uploading model files can take 1 to 2 hours. To expedite this process, you can submit a ticket to have the model files copied directly to your OSS bucket.

Run the following command to download the DeepSeek-R1-Distill-Qwen-7B model from ModelScope.
Note
Ensure that the git-lfs plugin is installed. If it is not, run yum install git-lfs or apt-get install git-lfs to install it. For more installation methods, see Installing Git Large File Storage.
```
git lfs install
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B.git
cd DeepSeek-R1-Distill-Qwen-7B/
git lfs pull
```

Create a directory in OSS and upload the model to OSS.

Note

For information about how to install and use ossutil, see Install ossutil.

ossutil mkdir oss://<your-bucket-name>/models/DeepSeek-R1-Distill-Qwen-7B
ossutil cp -r ./DeepSeek-R1-Distill-Qwen-7B oss://<your-bucket-name>/models/DeepSeek-R1-Distill-Qwen-7B

Create a persistent volume (PV) and a persistent volume claim (PVC). Configure a PV named llm-model and a PVC for the destination cluster. For more information, see Use a static OSS persistent volume.

The following table shows the basic configuration of the example PV.

Configuration item	Description
PV type	OSS
Name	llm-model
Access credentials	Configure the AccessKey ID and AccessKey secret used to access OSS.
Bucket ID	Select the OSS bucket created in the previous step.
OSS Path	Select the path where the model is located, such as `/models/DeepSeek-R1-Distill-Qwen-7B`.

The following table shows the basic configuration of the example PVC.

Configuration item	Description
Persistent Volume Claim Type	OSS
Name	llm-model
Allocation mode	Select Existing PV.
Existing PV	Click the link to select an existing PV and choose the created PV.

The following code provides an example YAML file.

apiVersion: v1
kind: Secret
metadata:
  name: oss-secret
stringData:
  akId: <your-oss-ak> # The AccessKey ID used to access OSS.
  akSecret: <your-oss-sk> # The AccessKey secret used to access OSS.
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: llm-model
  labels:
    alicloud-pvname: llm-model
spec:
  capacity:
    storage: 30Gi 
  accessModes:
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: ossplugin.csi.alibabacloud.com
    volumeHandle: llm-model
    nodePublishSecretRef:
      name: oss-secret
      namespace: default
    volumeAttributes:
      bucket: <your-bucket-name> # The name of the bucket.
      url: <your-bucket-endpoint> # The endpoint, such as oss-cn-hangzhou-internal.aliyuncs.com.
      otherOpts: "-o umask=022 -o max_stat_cache_size=0 -o allow_other"
      path: <your-model-path> # In this example, the path is /models/DeepSeek-R1-Distill-Qwen-7B/.
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: llm-model
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 30Gi
  selector:
    matchLabels:
      alicloud-pvname: llm-model

Step 2: Create a custom scheduling policy ResourcePolicy

Create a ResourcePolicy custom resource definition (CRD) to define scheduling rules for elastic resource priority. In this example, the `labelSelector` matches the `isvc.deepseek-predictor` application to define a rule. This rule specifies that the application is preferentially scheduled to the edge IDC resource pool. If the edge IDC resources are insufficient, the application is scheduled to virtual nodes in the cloud. For more information about how to use ResourcePolicy, see Customize elastic resource scheduling priority.

Important

When you create the application pod later, you must add a label that matches the `labelSelector` in the policy to associate the pod with this scheduling policy.

Create a ResourcePolicy CRD and save it as a file named nginx-resourcepolicy.yaml.

apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
  name: deepseek
  namespace: default
spec:
  selector:
    app: isvc.deepseek-predictor # This must be associated with the label of the pod that you create later.
  strategy: prefer
  units:
  - resource: ecs
    nodeSelector:
      alibabacloud.com/nodepool-id: np*********  # The ID of the edge node pool.
  - resource: eci

Deploy the custom scheduling policy in the cluster to define the scheduling priority.
```
kubectl create -f nginx-resourcepolicy.yaml
```

Step 3: Deploy the model

Query the status of the nodes in the cluster.

kubectl get nodes -owide

Expected output:

NAME                            STATUS   ROLES    AGE     VERSION            INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                              KERNEL-VERSION           CONTAINER-RUNTIME
cn-hangzhou.10.4.XX.25           Ready    <none>   10d     v1.30.7-aliyun.1   10.4.0.25     <none>        Alibaba Cloud Linux 3.2104 U11 (OpenAnolis Edition)   5.10.134-18.al8.x86_64   containerd://1.6.36
cn-hangzhou.10.4.XX.26           Ready    <none>   10d     v1.30.7-aliyun.1   10.4.0.26     <none>        Alibaba Cloud Linux 3.2104 U11 (OpenAnolis Edition)   5.10.134-18.al8.x86_64   containerd://1.6.36
idc001                           Ready    <none>   31s     v1.30.7-aliyun.1   10.4.0.185    <none>        Alibaba Cloud Linux 3.2104 U11 (OpenAnolis Edition)   5.10.134-18.al8.x86_64   containerd://1.6.36
virtual-kubelet-cn-hangzhou-b    Ready    agent    7d21h   v1.30.7-aliyun.1   10.4.0.180    <none>        <unknown>                                             <unknown>                <unknown>

The expected output shows that the cluster has one IDC node (idc001) and one virtual node (virtual-kubelet-cn-hangzhou-b). The IDC node has one V100 GPU card.

Deploy the DeepSeek model inference service using the vLLM model inference framework.

arena serve kserve \
    --name=deepseek \
    --annotation=k8s.aliyun.com/eci-use-specs=ecs.gn6e-c12g1.3xlarge \
    --annotation=k8s.aliyun.com/eci-vswitch=vsw-*********,vsw-********* \
    --image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm:v0.6.6 \
    --gpus=1 \
    --cpu=4 \
    --memory=12Gi \
    --scale-metric=DCGM_CUSTOM_PROCESS_SM_UTIL \
    --scale-target=50 \
    --min-replicas=1  \
    --max-replicas=3  \
    --data=llm-model:/model/DeepSeek-R1-Distill-Qwen-7B \
    "vllm serve /model/DeepSeek-R1-Distill-Qwen-7B --port 8080 --trust-remote-code --served-model-name deepseek-r1 --max-model-len 32768 --gpu-memory-utilization 0.95 --enforce-eager --dtype=half"

The key parameters are described in the following table.

Parameter	Description	Example
`--name`	The name of the inference service to submit. The name must be globally unique.	deepseek
`--image`	The registry address of the inference service. This example uses the vLLM inference framework.	kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm:v0.6.6
`--gpus`	The number of GPU cards required by the inference service. The default value is 0.	1
`--cpu`	The number of CPUs required by the inference service.	4
`--memory`	The amount of memory required by the inference service.	12Gi
`--scale-metric`	The metric for application auto scaling. This example uses the GPU utilization metric `DCGM_CUSTOM_PROCESS_SM_UTIL` for application scaling. For more metrics, see 2. Configure HPA.	DCGM_CUSTOM_PROCESS_SM_UTIL
`--scale-target`	The target value for application scaling. When GPU utilization exceeds 50%, replica scale-out begins.	50
`--min-replicas`	The minimum number of replicas.	1
`--max-replicas`	The maximum number of replicas.	3
`--data`	Specifies the model address for the service. In this example, the model is stored in `llm-model` and mounted to the `/mnt/models/` directory in the container.	llm-model:/model/DeepSeek-R1-Distill-Qwen-7B \ "vllm serve /model/DeepSeek-R1-Distill-Qwen-7B --port 8080 --trust-remote-code --served-model-name deepseek-r1 --max-model-len 32768 --gpu-memory-utilization 0.95 --enforce-eager --dtype=half"

Expected output:

WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/bingchang/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/bingchang/.kube/config
horizontalpodautoscaler.autoscaling/deepseek-hpa created
inferenceservice.serving.kserve.io/deepseek created
INFO[0002] The Job deepseek has been submitted successfully
INFO[0002] You can run `arena serve get deepseek --type kserve -n default` to check the job status

View the details of the inference service.

arena serve get deepseek

Expected output:

Name:       deepseek
Namespace:  default
Type:       KServe
Version:    1
Desired:    1
Available:  1
Age:        1m
Address:    http://deepseek-default.example.com
Port:       :80
GPU:        1


Instances:
  NAME                                 STATUS   AGE  READY  RESTARTS  GPU  NODE
  ----                                 ------   ---  -----  --------  ---  ----
  deepseek-predictor-6b9455f8c5-wl5lc  Running  1m   1/1    0         1    idc001

The output shows that the application pod of the inference service is scheduled to an IDC node, which is consistent with the custom scheduling priority.

Send the following request to the service to verify that the inference service is running correctly. You can obtain the request address from the details of the Ingress resource automatically created by KServe.

curl -H "Host: deepseek-default.example.com" -H "Content-Type: application/json" http://<idc-node-ip>:<ingress-svc-nodeport>/v1/chat/completions -d '{"model": "deepseek-r1", "messages": [{"role": "user", "content": "Say this is a test!"}], "max_tokens": 512, "temperature": 0.7, "top_p": 0.9, "seed": 10}'

Expected output:

{"id":"chatcmpl-efc1225ad2f33cc39a8ddbc4039a41b9","object":"chat.completion","created":1739861087,"model":"deepseek-r1","choices":[{"index":0,"message":{"role":"assistant","content":"Okay, so I need to figure out how to say \"This is a test!\" in Spanish. Hmm, I'm not super fluent in Spanish, but I know some basic phrases. Let me think about how to approach this.\n\nFirst, I remember that \"test\" is \"prueba\" in Spanish. So maybe I can start with \"Esto es una prueba.\" But I'm not sure if that's the best way to say it. Maybe there's a more common expression or a different structure.\n\nWait, I think there's a phrase that's commonly used in tests. Isn't it something like \"This is a test.\" or \"This is a quiz.\"? I think the Spanish equivalent would be \"Este es un test.\" That sounds more natural. Let me check if that makes sense.\n\nI can also think about how people use phrases in tests. Maybe they use \"This is the test\" or \"This is an exam.\" So perhaps \"Este es el test.\" or \"Este es el examen.\" I'm not sure which one is more appropriate.\n\nI should also consider the grammar. \"This is a test\" is a simple statement, so the subject is \"this\" (using \"este\"), the verb is \"is\" (using \"es\"), and the object is \"a test\" (using \"un test\"). So putting it together, it would be \"Este es un test.\"\n\nWait, but sometimes people use \"This is the test\" when referring to an important one, so maybe \"Este es el test.\" But I'm not entirely sure if that's the correct structure. Let me think about other similar phrases.\n\nI also recall that in some contexts, people might say \"This is a practice test\" or \"This is a sample test.\" But since the user just said \"This is a test,\" the most straightforward translation would be \"Este es un test.\"\n\nI should also consider if there are any idiomatic expressions or common phrases that are used in this context. For example, \"This is the test\" is often used to mean a significant exam or evaluation, so \"Este es el test\" might be more appropriate in that context.\n\nBut I'm a bit confused because I'm not 100% sure about the correct structure. Maybe I should look up some examples. Oh, wait, I can't look things up right now, so I'll have to rely on my memory.\n\nI think the basic structure is subject + verb + object. So \"this\" (this is \"este","tool_calls":[]},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":11,"total_tokens":523,"completion_tokens":512,"prompt_tokens_details":null},"prompt_logprobs":null}

Step 4: Simulate peak business requests to trigger cloud elasticity

Use the stress testing tool Hey to send many requests to the deployed inference service.

hey -z 5m -c 5 \
-m POST -host deepseek-default.example.com \
-H "Content-Type: application/json" \
-d '{"model": "deepseek-r1", "messages": [{"role": "user", "content": "Say this is a test!"}], "max_tokens": 512, "temperature": 0.7, "top_p": 0.9, "seed": 10}' \
http://<idc-node-ip>:<ingress-svc-nodeport>/v1/chat/completions

The preceding request is sent to the existing pod. The high volume of requests triggers a pod scale-out when the GPU utilization exceeds the 50% threshold.

View the details of the inference service.

arena serve get deepseek

Expected output:

Name:       deepseek
Namespace:  default
Type:       KServe
Version:    1
Desired:    3
Available:  2
Age:        18m
Address:    http://deepseek-default.example.com
Port:       :80
GPU:        3


Instances:
  NAME                                 STATUS   AGE  READY  RESTARTS  GPU  NODE
  ----                                 ------   ---  -----  --------  ---  ----
  deepseek-predictor-6b9455f8c5-dtzdv  Running  1m   0/1    0         1    virtual-kubelet-cn-hangzhou-h
  deepseek-predictor-6b9455f8c5-wl5lc  Running  18m  1/1    0         1    idc001
  deepseek-predictor-6b9455f8c5-zmpg8  Running  5m   1/1    0         1    virtual-kubelet-cn-hangzhou-h

At this point, two pod replicas for the inference job have been scaled out to the virtual node.