部署SGLang PD分离推理服务

部署PD分离架构推理服务。SGLang Prefill Server和Decode Server交互时序图如下所示。

• 收到用户推理请求后，Prefill Server将会创建一个Sender对象而Decode Server会创建一个Receiver对象。

• Prefill和Decode通过Handshake建立连接，Decode首先分配一块显存地址用于接收KVCache，Prefill Server完成计算后将KVCache传送给Decode Server，Decode Server收到KVCache后继续计算后续Token，直到完成用户的推理请求。

步骤一：准备Qwen3-32B模型文件

步骤二：部署SGLang PD分离架构的推理服务

本文使用RBG部署2P1D SGLang推理服务，部署架构图如下所示。

1. 创建sglang_pd.yaml文件。

   展开查看YAML代码示例。

   apiVersion: workloads.x-k8s.io/v1alpha1
   kind: RoleBasedGroup
   metadata:
     name: sglang-pd
   spec:
     roles:
       - name: scheduler
         replicas: 1
         dependencies: [ "decode", "prefill" ]
         template:
           spec:
             volumes:
               - name: model
                 persistentVolumeClaim:
                   claimName: llm-model
             containers:
               - name: scheduler
                 image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/anolis-docker-images/docker-temp:0.3.4.post2-sglang0.4.10.post2-pytorch2.7.1.8-cuda12.8.1-py312-alinux3.2104
                 command:
                   - sh
                   - -c
                   - python3 -m sglang.srt.disaggregation.mini_lb --prefill http://sglang-pd-prefill-0.sglang-pd-prefill:8000 http://sglang-pd-prefill-1.sglang-pd-prefill:8000 --prefill-bootstrap-ports 34000 34000 --decode http://sglang-pd-decode-0.sglang-pd-decode:8000 --host 0.0.0.0 --port 8000
                 volumeMounts:
                   - mountPath: /models/Qwen3-32B/
                     name: model
   
       - name: prefill
         replicas: 2
         template:
           metadata:
             labels:
               alibabacloud.com/inference-workload: sglang-pd-prefill
               alibabacloud.com/inference_backend: sglang
           spec:
             volumes:
               - name: model
                 persistentVolumeClaim:
                   claimName: llm-model
               - name: dshm
                 emptyDir:
                   medium: Memory
                   sizeLimit: 15Gi
             containers:
               - name: sglang-prefill
                 image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/anolis-docker-images/docker-temp:0.3.4.post2-sglang0.4.10.post2-pytorch2.7.1.8-cuda12.8.1-py312-alinux3.2104
                 imagePullPolicy: Always
                 env:
                   - name: POD_IP
                     valueFrom:
                       fieldRef:
                         fieldPath: status.podIP
                 command:
                   - sh
                   - -c
                   - python3 -m sglang.launch_server --tp 2 --model-path /models/Qwen3-32B/ --disaggregation-mode prefill --port 8000 --disaggregation-bootstrap-port 34000 --host $(POD_IP) --enable-metrics
                 ports:
                   - containerPort: 8000
                     name: http
                   - containerPort: 34000
                     name: bootstrap
                 readinessProbe:
                   initialDelaySeconds: 30
                   periodSeconds: 10
                   tcpSocket:
                     port: 8000
                 resources:
                   limits:
                     nvidia.com/gpu: "2"
                     aliyun/erdma: 1
                     memory: "16Gi"
                     cpu: "4"
                   requests:
                     nvidia.com/gpu: "2"
                     aliyun/erdma: 1
                     memory: "16Gi"
                     cpu: "4"
                 volumeMounts:
                   - mountPath: /models/Qwen3-32B/
                     name: model
                   - mountPath: /dev/shm
                     name: dshm
   
       - name: decode
         replicas: 1
         template:
           metadata:
             labels:
               alibabacloud.com/inference-workload: sglang-pd-decode
               alibabacloud.com/inference_backend: sglang
           spec:
             volumes:
               - name: model
                 persistentVolumeClaim:
                   claimName: llm-model
               - name: dshm
                 emptyDir:
                   medium: Memory
                   sizeLimit: 15Gi
             containers:
               - name: sglang-decode
                 image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/anolis-docker-images/docker-temp:0.3.4.post2-sglang0.4.10.post2-pytorch2.7.1.8-cuda12.8.1-py312-alinux3.2104
                 imagePullPolicy: Always
                 env:
                   - name: POD_IP
                     valueFrom:
                       fieldRef:
                         fieldPath: status.podIP
                 command:
                   - sh
                   - -c
                   - python3 -m sglang.launch_server --tp 2 --model-path /models/Qwen3-32B/ --disaggregation-mode decode --port 8000 --host $(POD_IP) --enable-metrics
                 ports:
                   - containerPort: 8000
                     name: http
                 readinessProbe:
                   initialDelaySeconds: 30
                   periodSeconds: 10
                   tcpSocket:
                     port: 8000
                 resources:
                   limits:
                     nvidia.com/gpu: "2"
                     aliyun/erdma: 1
                     memory: "16Gi"
                     cpu: "4"
                   requests:
                     nvidia.com/gpu: "2"
                     aliyun/erdma: 1
                     memory: "16Gi"
                     cpu: "4"
                 volumeMounts:
                   - mountPath: /models/Qwen3-32B/
                     name: model
                   - mountPath: /dev/shm
                     name: dshm
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: sglang-pd
     name: sglang-pd
     namespace: default
   spec:
     ports:
       - name: http
         port: 8000
         protocol: TCP
         targetPort: 8000
     selector:
       rolebasedgroup.workloads.x-k8s.io/name: sglang-pd
       rolebasedgroup.workloads.x-k8s.io/role: scheduler
     type: ClusterIP
   

2. 部署SGLang PD分离推理服务。

   kubectl create -f sglang_pd.yaml

步骤三：验证推理服务

1. 执行以下命令，在推理服务与本地环境之间建立端口转发。

   重要

   kubectl port-forward建立的端口转发不具备生产级别的可靠性、安全性和扩展性，因此仅适用于开发和调试目的，不适合在生产环境使用。更多关于Kubernetes集群内生产可用的网络方案的信息，请参见Ingress管理。

   kubectl port-forward svc/sglang-pd 8000:8000

   预期输出：

   Forwarding from 127.0.0.1:8000 -> 8000
   Forwarding from [::1]:8000 -> 8000

2. 执行以下命令，向模型推理服务发送了一条示例的模型推理请求。

   curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json"  -d '{"model": "/models/Qwen3-32B", "messages": [{"role": "user", "content": "测试一下"}], "max_tokens": 30, "temperature": 0.7, "top_p": 0.9, "seed": 10}'

   预期输出：

   {"id":"29f3fdac693540bfa7808fc1a8701758","object":"chat.completion","created":1753695366,"model":"/models/Qwen3-32B","choices":[{"index":0,"message":{"role":"assistant","content":"<think>\n好的，用户让我测试一下，我需要先确认他们的具体需求。可能他们想测试我的功能，比如回答问题、生成内容","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"length","matched_stop":null}],"usage":{"prompt_tokens":10,"total_tokens":40,"completion_tokens":30,"prompt_tokens_details":null}}

   输出结果表明模型可以根据给定的输入（在这个例子中是一条测试消息）生成相应的回复。