使用ACS GPU算力构建LLM推理服务

部署大模型推理服务

1. 根据以下内容创建PVC。具体操作，请参见创建PVC。

   参数	示例
   名称	nas-test。
   分配模式	选择使用挂载点域名创建。
   总量	20Gi。
   挂载点域名	08cxxxxxxec-wxxxxxxcn-hangzhou.nas.aliyuncs.com

2. 使用kubectl连接到ACS集群。具体操作，请参见获取集群KubeConfig并通过kubectl工具连接集群。

3. 使用以下内容，创建your-deployment.yaml。

   说明

   关于以下示例中使用的GPU卡具体型号，请参见GPU型号及驱动版本。

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: my-deployment
     name: my-deployment
     namespace: default
   spec:
     progressDeadlineSeconds: 600
     replicas: 1
     revisionHistoryLimit: 10
     selector:
       matchLabels:
         app: my-deployment
     template:
       metadata:
         labels:
           # 指定GPU型号为example-model，请按实际情况填写，如T4
           alibabacloud.com/gpu-model-series: example-model
           # 指定compute-class为gpu类型
           alibabacloud.com/compute-class: gpu
           app: my-deployment
       spec:
         containers:
           - command:
             - sh
             - -c
             - python3 -m vllm.entrypoints.openai.api_server --model /mnt/Qwen2.5-7B-Instruct --trust-remote-code --tensor-parallel-size 1  --disable-custom-all-reduce
             image: egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/llm-inference:vllm0.5.4-deepgpu-llm24.7-pytorch2.4.0-cuda12.4-ubuntu22.04
             imagePullPolicy: IfNotPresent #Always
             name: my-deployment
             resources:
               limits:
                 cpu: 16
                 memory: 64Gi
                 nvidia.com/gpu: "1"
               requests:
                 cpu: 16
                 memory: 64Gi
                 nvidia.com/gpu: "1"
             terminationMessagePath: /dev/termination-log
             terminationMessagePolicy: File
             volumeMounts:
               - mountPath: /mnt
                 name: nas-test
               - mountPath: /dev/shm
                 name: cache-volume
         dnsPolicy: ClusterFirst
         restartPolicy: Always
         schedulerName: default-scheduler
         securityContext: {}
         terminationGracePeriodSeconds: 30
         volumes:
           - name: nas-test
             persistentVolumeClaim:
               # PVC的声明
               claimName: nas-test
           - name: cache-volume
             emptyDir:
               medium: Memory
               sizeLimit: 64G
   ---
   apiVersion: v1
   kind: Service
   metadata:
     annotations:
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "internet"
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: ipv4
     labels:
       app: my-deployment
     name: svc-llm
     namespace: default
   spec:
     externalTrafficPolicy: Local
     ports:
     - name: serving
       port: 8000
       protocol: TCP
       targetPort: 8000
     selector:
       app: my-deployment
     type: LoadBalancer

4. 执行以下命令，部署大模型推理任务和对应的service。

   kubectl apply -f your-deployment.yaml

5. 由于镜像较大，您可以通过执行以下命令持续关注模型部署情况，大约需要20分钟左右。

   kubectl logs -f my-deployment-787b8xxxxx-xxxxx

   当出现以下格式的输出时，说明模型已经部署成功。

   INFO:     Started server process [2]
   INFO:     Waiting for application startup.
   INFO:     Application startup complete.
   INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
   INFO 12-13 12:39:11 metrics.py:406] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
   INFO 12-13 12:39:21 metrics.py:406] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
   INFO 12-13 12:39:31 metrics.py:406] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
   INFO 12-13 12:39:41 metrics.py:406] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.

6. 执行以下命令，查看部署的Service。

   kubectl get svc

   预期输出：

   NAME         TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)    AGE
   kubernetes   ClusterIP      10.0.0.1       <none>           443/TCP    3h38m
   svc-llm      LoadBalancer   10.0.143.103   112.xxx.xxx.177  8000/TCP   58s

   其中EXTERNAL-IP为Service对外公开的IP。记录此IP，方便后续进行测试。