部署Dynamo PD分离推理服务

部署Dynamo PD分离架构推理服务。Dynamo PD分离时序图如下所示。

• 用户请求首先发送给processor组件，经由router选出合适的Decode Worker，将请求转发给Decode Worker。

• 由Decode Worker判断是否prefill计算是在本地完成还是远程完成。如果需要远程计算，则向PrefillQueue中发送请求。

• PrefillWorker从Queue获取请求后进行Prefill计算。等计算完成后，将Prefill KVCache传输给Decode Worker。

步骤一：准备Qwen3-32B模型文件

步骤二：安装ETCD及NATS服务

Dynamo框架中跨节点通信使用的是NIXL。NIXL启动时会向ETCD注册，从而实现相互发现。NATS服务主要用于Prefill和Decode Worker间消息传递。因此部署推理服务前，需要先部署ETCD及NATS服务。

1. 执行以下命令创建etcd.yaml并部署ETCD服务。

   展开查看相关YAML示例。

   apiVersion: v1
   kind: Service
   metadata:
     name: etcd
     labels:
       app: etcd
   spec:
     ports:
       - port: 2379
         name: client
       - port: 2380
         name: peer
     clusterIP: None # Enables headless service mode
     selector:
       app: etcd
   ---
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: etcd
     labels:
       app: etcd
   spec:
     selector:
       matchLabels:
         app: etcd
     serviceName: "etcd"
     replicas: 1
     template:
       metadata:
         labels:
           app: etcd
       spec:
         containers:
           - name: etcd
             # image: bitnami/etcd:3.5.19
             image: ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/etcd:3.6.1
             volumeMounts:
               - name: data
                 mountPath: /var/lib/etcd
             env:
               - name: ETCDCTL_API
                 value: "3"
               - name: ALLOW_NONE_AUTHENTICATION
                 value: "yes"
         volumes:
           - name: data
             emptyDir: {}

   kubectl apply -f etcd.yaml

2. 执行以下命令创建nats.yaml并部署NATS服务。

   展开查看相关YAML示例。

   apiVersion: v1
   kind: Service
   metadata:
     name: nats
     labels:
       app: nats
   spec:
     ports:
       - port: 4222
         name: client
       - port: 8222
         name: management
       - port: 6222
         name: cluster
     selector:
       app: nats
     type: ClusterIP
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nats
     labels:
       app: nats
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: nats
     template:
       metadata:
         labels:
           app: nats
       spec:
         containers:
           - name: nats
             # image: nats:latest
             image: ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/nats:2.11.5
             args:
               - -js
               - --trace
               - -m
               - "8222"
             ports:
               - containerPort: 4222
               - containerPort: 8222
               - containerPort: 6222

   kubectl apply -f nats.yaml

步骤三：部署Dynamo PD分离架构的推理服务

本文使用RBG部署2P1D Dynamo推理服务，部署架构图如下所示。 Prefill和Decode均采用了TP=2的方式部署。

1. 使用dynamo-configs.yaml配置qwen3模型及Dynamo框架相关的配置文件。

   展开查看相关YAML示例。

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: dynamo-configs
   data:
     pd_disagg.py: |
       from components.frontend import Frontend
       from components.kv_router import Router
       from components.processor import Processor
   
       Frontend.link(Processor).link(Router)
   
     qwen3.yaml: |
       Common:
         model: /models/Qwen3-32B/
         kv-transfer-config: '{"kv_connector":"DynamoNixlConnector"}'
         router: round-robin
         # Number of tokens in a batch for more efficient chunked transfers to GPUs.
         block-size: 128
         max-model-len: 2048
         max-num-batched-tokens: 2048
         disable-log-requests: true
   
       Frontend:
         served_model_name: qwen
         endpoint: dynamo.Processor.chat/completions
         port: 8000
   
       Processor:
         common-configs: [ model, router ]
   
       VllmWorker:
         common-configs: [ model, kv-transfer-config, router, block-size, max-model-len, disable-log-requests ]
         # Enable prefill at different workers.
         remote-prefill: true
         # Disable local prefill so only disaggregated prefill is used.
         conditional-disagg: false
         gpu-memory-utilization: 0.95
         tensor-parallel-size: 1
         ServiceArgs:
           workers: 1
           resources:
             gpu: 1
   
       PrefillWorker:
         common-configs: [ model, kv-transfer-config, block-size, max-model-len, max-num-batched-tokens, gpu-memory-utilization, disable-log-requests ]
         tensor-parallel-size: 1
         gpu-memory-utilization: 0.95
         ServiceArgs:
           workers: 1
           resources:
             gpu: 1
   

   kubectl apply -f dynamo-configs.yaml

2. 准备Dynamo Runtime镜像。

   具体操作，请参见Dynamo社区文档获取，或制作使用vllm作为推理框架的Dynamo Runtime容器镜像。

3. 执行以下命令创建dynamo.yaml并部署服务。

   展开查看相关YAML示例。

   apiVersion: workloads.x-k8s.io/v1alpha1
   kind: RoleBasedGroup
   metadata:
     name: dynamo-pd
     namespace: default
   spec:
     roles:
       - name: processor
         replicas: 1
         template:
           spec:
             containers:
               - command:
                   - sh
                   - -c
                   - cd /workspace/examples/llm; dynamo serve graphs.pd_disagg:Frontend -f ./configs/qwen3.yaml
                 env:
                   - name: DYNAMO_NAME
                     value: dynamo
                   - name: DYNAMO_NAMESPACE
                     value: default
                   - name: ETCD_ENDPOINTS
                     value: http://etcd:2379
                   - name: NATS_SERVER
                     value: nats://nats:4222
                   - name: DYNAMO_RP_TIMEOUT
                     value: "60"
                 image: #步骤2中所构建的Dynamo Runtime镜像地址
                 name: processor
                 ports:
                   - containerPort: 8000
                     name: health
                     protocol: TCP
                   - containerPort: 9345
                     name: request
                     protocol: TCP
                   - containerPort: 443
                     name: api
                     protocol: TCP
                   - containerPort: 9347
                     name: metrics
                     protocol: TCP
                 readinessProbe:
                   initialDelaySeconds: 30
                   periodSeconds: 30
                   tcpSocket:
                     port: 8000
                 resources:
                   limits:
                     cpu: "8"
                     memory: 12Gi
                   requests:
                     cpu: "8"
                     memory: 12Gi
                 volumeMounts:
                   - mountPath: /models/Qwen3-32B/
                     name: model
                   - mountPath: /workspace/examples/llm/configs/qwen3.yaml
                     name: dynamo-configs
                     subPath: qwen3.yaml
                   - mountPath: /workspace/examples/llm/graphs/pd_disagg.py
                     name: dynamo-configs
                     subPath: pd_disagg.py
             volumes:
               - name: model
                 persistentVolumeClaim:
                   claimName: llm-model
               - configMap:
                   name: dynamo-configs
                 name: dynamo-configs
       - name: prefill
         replicas: 2
         template:
           spec:
             containers:
               - command:
                   - sh
                   - -c
                   - cd /workspace/examples/llm; dynamo serve components.prefill_worker:PrefillWorker -f ./configs/qwen3.yaml
                 env:
                   - name: DYNAMO_NAME
                     value: dynamo
                   - name: DYNAMO_NAMESPACE
                     value: default
                   - name: ETCD_ENDPOINTS
                     value: http://etcd:2379
                   - name: NATS_SERVER
                     value: nats://nats:4222
                   - name: DYNAMO_RP_TIMEOUT
                     value: "60"
                 image: #步骤2中所构建的Dynamo Runtime镜像地址
                 name: prefill-worker
                 resources:
                   limits:
                     cpu: "12"
                     memory: 50Gi
                     nvidia.com/gpu: "2"
                   requests:
                     cpu: "12"
                     memory: 50Gi
                     nvidia.com/gpu: "2"
                 volumeMounts:
                   - mountPath: /models/Qwen3-32B/
                     name: model
                   - mountPath: /workspace/examples/llm/configs/qwen3.yaml
                     name: dynamo-configs
                     subPath: qwen3.yaml
             volumes:
               - name: model
                 persistentVolumeClaim:
                   claimName: llm-model
               - configMap:
                   name: dynamo-configs
                 name: dynamo-configs
       - name: decoder
         replicas: 1
         template:
           spec:
             containers:
               - command:
                   - sh
                   - -c
                   - cd /workspace/examples/llm; dynamo serve components.worker:VllmWorker -f ./configs/qwen3.yaml --service-name VllmWorker
                 env:
                   - name: DYNAMO_NAME
                     value: dynamo
                   - name: DYNAMO_NAMESPACE
                     value: default
                   - name: ETCD_ENDPOINTS
                     value: http://etcd:2379
                   - name: NATS_SERVER
                     value: nats://nats:4222
                   - name: DYNAMO_RP_TIMEOUT
                     value: "60"
                 image: #步骤2中所构建的Dynamo Runtime镜像地址
                 name: vllm-worker
                 resources:
                   limits:
                     cpu: "12"
                     memory: 50Gi
                     nvidia.com/gpu: "2"
                   requests:
                     cpu: "12"
                     memory: 50Gi
                     nvidia.com/gpu: "2"
                 volumeMounts:
                   - mountPath: /models/Qwen3-32B/
                     name: model
                   - mountPath: /workspace/examples/llm/configs/qwen3.yaml
                     name: dynamo-configs
                     subPath: qwen3.yaml
             volumes:
               - name: model
                 persistentVolumeClaim:
                   claimName: llm-model
               - configMap:
                   name: dynamo-configs
                 name: dynamo-configs
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: dynamo-service
   spec:
     type: ClusterIP
     ports:
       - port: 8000
         protocol: TCP
         targetPort: 8000
     selector:
       rolebasedgroup.workloads.x-k8s.io/name: dynamo-pd
       rolebasedgroup.workloads.x-k8s.io/role: processor

   kubectl apply -f ./dynamo.yaml

步骤四：验证推理服务

1. 执行以下命令，在推理服务与本地环境之间建立端口转发。

   重要

   kubectl port-forward建立的端口转发不具备生产级别的可靠性、安全性和扩展性，因此仅适用于开发和调试目的，不适合在生产环境使用。更多关于Kubernetes集群内生产可用的网络方案的信息，请参见Ingress管理。

   kubectl port-forward svc/dynamo-service 8000:8000

   预期输出：

   Forwarding from 127.0.0.1:8000 -> 8000
   Forwarding from [::1]:8000 -> 8000

2. 执行以下命令，向模型推理服务发送了一条示例的模型推理请求。

   curl http://localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{"model": "qwen","messages": [{"role": "user","content": "测试一下"}],"stream":false,"max_tokens": 30}'

   预期输出：

   {"id":"31ac3203-c5f9-4b06-a4cd-4435a78d3b35","choices":[{"index":0,"message":{"content":"<think>\n好的，用户发来“测试一下”，我需要先确认他们的意图。可能是在测试我的反应速度或者功能，也可能是想","refusal":null,"tool_calls":null,"role":"assistant","function_call":null,"audio":null},"finish_reason":"length","logprobs":null}],"created":1753702438,"model":"qwen","service_tier":null,"system_fingerprint":null,"object":"chat.completion","usage":null}

   输出结果表明模型可以根据给定的输入（在这个例子中是一条测试消息）生成相应的回复。