Gateway with Inference Extension相关组件的配置说明

ACKTrafficFilter

ACKTrafficFilter能够为网关路由提供扩展的流量处理选项，如对接阿里云内容安全实现AI内容安全审查。

以下示例通过结合 ACKTrafficFilter 和 HTTPRoute，为网关接入阿里云内容安全服务，实现 AI 内容安全审查的能力，请参考使用Gateway with Inference Extension实现AI内容审查。

apiVersion: inferenceextension.alibabacloud.com/v1alpha1
kind: ACKTrafficFilter
metadata:
  name: aisg
spec:
  aiContentSecurity:
    accessKey: XXXXXX
    secretKey: XXXXXX
    aliyunEndpoint: green-cip-vpc.cn-hangzhou.aliyuncs.com
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mock-route
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: mock-gateway
    sectionName: llm-gw
  rules:
  - backendRefs:
    - group: inference.networking.x-k8s.io
      kind: InferencePool
      name: mock-pool
    filters:
    - type: ExtensionRef
      extensionRef:
        group: inferenceextension.alibabacloud.com
        kind: ACKTrafficFilter
        name: aisg
    matches:
    - path:
        type: PathPrefix
        value: /

配置项说明

ACKTrafficFilterSpec

AIContentSecurity

ExtensionDeploymentOverlay

InferenceTrafficPolicy

InferenceTrafficPolicy用于配置针对指定InferencePool内的推理工作负载进行路由时，Gateway with Inference Extension采用的路由策略，包括推理工作负载的部署形态信息、使用的路由算法及其参数等。

默认情况下，Gateway with Inference Extension将针对InferencePool后端的单机vLLM推理服务，采用基于KVCache利用率与排队请求数的智能路由负载均衡策略进行请求转发，具体内容参考使用Gateway with Inference Extension实现智能路由与流量管理。当推理服务使用其他部署形态、或需要调整Gateway with Inference Extension的路由算法以及相关算法参数时，可使用InferenceTrafficPolicy对智能路由策略进行调整。

示例一：对单机SGLang推理服务配置前缀感知的负载均衡（估算模式）

该策略针对同命名空间下名为 qwen-inference-pool 的InferencePool生效。

apiVersion: inferenceextension.alibabacloud.com/v1alpha1
kind: InferenceTrafficPolicy
metadata:
  name: sglang-prefixcache-policy
spec:
  poolRef:
    name: qwen-inference-pool
  modelServerRuntime: sglang
  profile:
    single:
      trafficPolicy:
        prefixCache:
          mode: estimate

示例二：对单机vLLM推理服务配置前缀感知的负载均衡（跟踪模式）

apiVersion: inferenceextension.alibabacloud.com/v1alpha1
kind: InferenceTrafficPolicy
metadata:
  name: inference-policy
spec:
  poolRef:
    name: qwen-inference-pool
  profile:
    single:
      trafficPolicy:
        prefixCache:
          mode: tracking
          trackingConfig:
            indexerConfig:
              tokenProcessorConfig:
                blockSize: 64
                hashSeed: 42
                model: Qwen/Qwen3-32B
            kvEventsConfig:
              zmqEndpoint: tcp://*:5557
              topicFilter: kv@

示例三：对SGLang PD分离推理服务配置路由

apiVersion: inferenceextension.alibabacloud.com/v1alpha1
kind: InferenceTrafficPolicy
metadata:
  name: inference-policy
spec:
  poolRef:
    name: qwen-inference-pool
  modelServerRuntime: sglang
  profile:
    pd:
      pdRoleLabelName: rolebasedgroup.workloads.x-k8s.io/role
      kvTransfer:
        bootstrapPort: 34000

配置项说明

spec

PoolObjectReference

DeploymentProfile

SingleProfile

PDProfile

KVTransfer

TrafficPolicy

PrefixCachePolicy

EstimateConfig

TrackingConfig

IndexerConfig

TokenProcessorConfig

KvBlockIndexConfig

KVEventsConfig

BurstLimit