快速体验Gateway with Inference Extension智能推理路由_容器计算服务(ACS)-阿里云帮助中心

LLM应用通常需要使用GPU来启动，而且GPU类型的节点或虚拟节点较CPU节点有着较高的开销。为此，Gateway with Inference Extension组件团队提供了一个使用CPU算力快速体验大语言模型（LLM）推理场景的智能负载均衡能力的方式。本文介绍如何基于Gateway with Inference Extension构建一个mock环境来快速体验推理服务的智能负载均衡能力。

前提条件

已安装Gateway with Inference Extension并勾选启用Gateway API推理扩展。操作入口，请参见安装组件。

重要

本文构建的mock环境仅适用于快速体验Gateway with Inference Extension组件的一些基础AI能力，例如灰度发布、请求熔断、流量镜像等，不适用于需要进行压测的测试场景，也不建议在生产环境中使用。

操作步骤

步骤一：部署mock模型示例应用

创建mock-vllm.yaml。

apiVersion: v1
kind: ServiceAccount
metadata:
  name: mock-vllm
---
apiVersion: v1
kind: Service
metadata:
  name: mock-vllm
  labels:
    app: mock-vllm
    service: mock-vllm
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 8000
  selector:
    app: mock-vllm
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mock-vllm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mock-vllm
  template:
    metadata:
      labels:
        app: mock-vllm
    spec:
      serviceAccountName: mock-vllm
      containers:
      - image: registry-cn-hangzhou.ack.aliyuncs.com/dev/mock-vllm:v0.1.7-g3cffa27-aliyun
        imagePullPolicy: IfNotPresent
        name: mock-vllm
        ports:
        - containerPort: 8000

部署示例应用。
```
kubectl apply -f mock-vllm.yaml
```

创建sleep.yaml。

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image:  registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/curl:asm-sleep
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true

部署sleep应用，用于后续对示例应用发起测试请求。
```
kubectl apply -f sleep.yaml
```

步骤二：部署inference资源

创建inference-rule.yaml。

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: mock-pool
spec:
  extensionRef:
    group: ""
    kind: Service
    name: mock-ext-proc
  selector:
    app: mock-vllm
  targetPortNumber: 8000
---
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: mock-model
spec:
  criticality: Critical
  modelName: mock
  poolRef:
    group: inference.networking.x-k8s.io
    kind: InferencePool
    name: mock-pool
  targetModels:
  - name: mock
    weight: 100

部署inferencePool和inferenceModel。
```
kubectl apply -f inference-rule.yaml
```

步骤三：部署网关和路由规则

创建gateway.yaml。

kind: GatewayClass
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: inference-gateway
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: mock-gateway
spec:
  gatewayClassName: inference-gateway
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: custom-proxy-config
  listeners:
    - name: llm-gw
      protocol: HTTP
      port: 80
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: ClusterIP
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: mock-client-buffer-limit
spec:
  connection:
    bufferLimit: 20Mi
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: mock-gateway
---

创建httproute.yaml。

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mock-route
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: mock-gateway
    sectionName: llm-gw
  rules:
  - backendRefs:
    - group: inference.networking.x-k8s.io
      kind: InferencePool
      name: mock-pool
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /

部署网关和路由规则。

kubectl apply -f gateway.yaml
kubectl apply -f httproute.yaml

步骤四：发起测试

获取网关IP。

export GATEWAY_ADDRESS=$(kubectl get gateway/mock-gateway -o jsonpath='{.status.addresses[0].value}')
echo ${GATEWAY_ADDRESS}

从sleep应用中发起访问。

kubectl exec deployment/sleep -it -- curl -X POST ${GATEWAY_ADDRESS}/v1/chat/completions \
  -H 'Content-Type: application/json' -H "host: example.com" -v -d '{
    "model": "mock",
    "max_completion_tokens": 100,
    "temperature": 0,
    "messages": [
      {
        "role": "user",
        "content": "introduce yourself"
      }
    ]
}'

预期输出：

*   Trying 192.168.12.230:80...
* Connected to 192.168.12.230 (192.168.12.230) port 80
> POST /v1/chat/completions HTTP/1.1
> Host: example.com
> User-Agent: curl/8.8.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 184
> 
* upload completely sent off: 184 bytes
< HTTP/1.1 200 OK
< date: Tue, 27 May 2025 08:21:37 GMT
< server: uvicorn
< content-length: 354
< content-type: application/json
< 
* Connection #0 to host 192.168.12.230 left intact
{"id":"3bcc1fdd-e514-4a06-95aa-36c904015639","object":"chat.completion","created":1748334097.297188,"model":"mock","choices":[{"index":"0","message":{"role":"assistant","content":"As a mock AI Assitant, I can only echo your last message: introduce yourself"},"finish_reason":"stop"}],"usage":{"prompt_tokens":18,"completion_tokens":76,"total_tokens":94}}

步骤五：清理环境

若您不再需要使用此环境，您可以将本文中创建的所有YAML文件移动在一个新的目录下，执行以下命令进行清理：

kubectl delete -f .