Use Gateway with Inference Extension for AI content moderation

Step 1: Install the Gateway with Inference Extension plug-in

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Add-ons.

3. On the Component Management page, search for the Gateway with Inference Extension and click Install on the component card. In the dialog box that appears, select Enable Gateway API Inference Extension and follow the prompts to complete the installation.

Step 2: Install the ACKTrafficFilter plug-in service

1. Activate the Alibaba Cloud Content Moderation service.

2. Replace <ALIYUN_ACCESS_KEY_ID> and <ALIYUN_ACCESS_KEY_SECRET> with the AccessKey ID and AccessKey secret of the destination account. Replace <ENDPOINT> with the internal network endpoint for the cluster's region from the table below. If the destination region is not in the table, use the public endpoint of the nearest region. Then, save the content as a file named acktrafficfilter.yaml.

   If you use the AccessKey of a Resource Access Management (RAM) user, make sure that the AliyunYundunGreenWebFullAccess policy is granted to the user.

   apiVersion: inferenceextension.alibabacloud.com/v1alpha1
   kind: ACKTrafficFilter
   metadata:
     name: content-security-filter
   spec:
     aiContentSecurity:
       accessKey: <ALIYUN_ACCESS_KEY_ID>   
       secretKey: <ALIYUN_ACCESS_KEY_SECRET>
       aliyunEndpoint: <ENDPOINT>

   Region	Public endpoint	Internal endpoint
   China (Beijing)	green-cip.cn-beijing.aliyuncs.com	green-cip-vpc.cn-beijing.aliyuncs.com
   China (Shanghai)	green-cip.cn-shanghai.aliyuncs.com	green-cip-vpc.cn-shanghai.aliyuncs.com
   China (Hangzhou)	green-cip.cn-hangzhou.aliyuncs.com	green-cip-vpc.cn-hangzhou.aliyuncs.com
   China (Shenzhen)	green-cip.cn-shenzhen.aliyuncs.com	green-cip-vpc.cn-shenzhen.aliyuncs.com
   China (Chengdu)	green-cip.cn-chengdu.aliyuncs.com	Not available

3. Create the acktrafficfilter plug-in.

   kubectl apply -f acktrafficfilter.yaml

Step 3: Verify the content moderation

The following steps show how to create a gateway named mock-gateway, deploy a sample application that simulates a large language model, configure a forwarding route for the application, and send a test request.

1. Create the backend AI service. Save the following YAML content as a file named mock-vllm.yaml. Then, run the kubectl apply -f mock-vllm.yaml command.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: mock-vllm
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: mock-vllm
     template:
       metadata:
         labels:
           app: mock-vllm
       spec:
         containers:
         - args:
           - --model
           - mock
           - --port
           - "8000"
           image: registry-cn-hangzhou.ack.aliyuncs.com/dev/mock-vllm:v0.2.3-g053679d-aliyun
           imagePullPolicy: IfNotPresent
           name: vllm-sim
           ports:
           - containerPort: 8000
             name: http
             protocol: TCP
   ---
   apiVersion: inference.networking.x-k8s.io/v1alpha2
   kind: InferencePool
   metadata:
     name: mock-pool
   spec:
     extensionRef:
       group: ""
       kind: Service
       name: mock-ext-proc
     selector:
       app: mock-vllm
     targetPortNumber: 8000
   ---
   apiVersion: inference.networking.x-k8s.io/v1alpha2
   kind: InferenceModel
   metadata:
     name: mock-model
   spec:
     criticality: Critical
     modelName: mock
     poolRef:
       group: inference.networking.x-k8s.io
       kind: InferencePool
       name: mock-pool
     targetModels:
     - name: mock
       weight: 100

2. Create the gateway and configure route forwarding. Save the following YAML content as a file named mock-gateway.yaml. Then, run the kubectl apply -f mock-gateway.yaml command.

   apiVersion: gateway.networking.k8s.io/v1
   kind: Gateway
   metadata:
     name: mock-gateway
   spec:
     gatewayClassName: ack-gateway
     listeners:
     - name: llm-gw
       protocol: HTTP
       port: 8080
   ---
   apiVersion: gateway.networking.k8s.io/v1
   kind: HTTPRoute
   metadata:
     name: mock-route
   spec:
     parentRefs:
     - group: gateway.networking.k8s.io
       kind: Gateway
       name: mock-gateway
       sectionName: llm-gw
     rules:
     - backendRefs:
       - group: inference.networking.x-k8s.io
         kind: InferencePool
         name: mock-pool
       filters:
       - type: ExtensionRef
         extensionRef:
           group: inferenceextension.alibabacloud.com
           kind: ACKTrafficFilter
           name: content-security-filter  # Reference the ACKTrafficFilter instance
       matches:
       - path:
           type: PathPrefix
           value: /

3. Modify <Request Content> and send a request to verify the content moderation.

   export GATEWAY_ADDRESS=$(kubectl get gateway/mock-gateway -o jsonpath='{.status.addresses[0].value}')
   curl -X POST ${GATEWAY_ADDRESS}:8080/v1/chat/completions \
     -H 'Content-Type: application/json' -H "host: example.com" -v -d '{
       "model": "mock",
       "max_completion_tokens": 100,
       "temperature": 0,
       "messages": [
         {
           "role": "user",
           "content": "<Request Content>"
         }
       ]
   }'

   Output for a passed review

   {
       "id": "chatcmpl-9bffeb49-057e-4c42-97e5-6fd62e3e996e",
       "created": 1759057516,
       "model": "mock",
       "usage": {
           "prompt_tokens": 1,
           "completion_tokens": 7,
           "total_tokens": 8
       },
       "object": "chat.completion",
       "choices": [
           {
               "index": 0,
               "finish_reason": "stop",
               "message": {
                   "role": "assistant",
                   "content": "Today is a nice sunny day."
               }
           }
       ]
   }

   Output for a failed review

   The HTTP response status is 403 Forbidden, and the value of the content field in the response is replaced with a safety notice.

   {
       "id": "chatcmpl-uuFxPpxMo63DFt4lxiUjFY70kwgVs",
       "object": "chat.completion",
       "model": "from-security-guard",
       "choices": [
           {
               "index": 0,
               "message": {
                   "role": "assistant",
                   "content": "I am sorry, but as an AI language model, I cannot participate in or promote discussions about pornographic content. I can only provide answers to legal and ethical questions related to topics such as education, culture, and history. If you need help with other questions, please let me know."
               },
               "logprobs": null,
               "finish_reason": "stop"
           }
       ],
       "usage": {
           "prompt_tokens": 0,
           "completion_tokens": 0,
           "total_tokens": 0
       }
   }

Step 4: Clean up the environment

• Clean up cluster resources:

  # Delete the gateway and route
  kubectl delete -f mock-gateway.yaml
  # Delete the content moderation plug-in
  kubectl delete -f acktrafficfilter.yaml
  # Delete the backend application
  kubectl delete -f mock-vllm.yaml

• On the Add-ons page, search for Gateway with Inference Extension. On the component card, click Uninstall.