Protect user data with the ASM LLMProxy plugin

Step 1: Deploy the client application

Use kubectl to connect to the Kubernetes cluster that you added to the ASM instance. Create a file named sleep.yaml with the following content.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: registry.cn-hangzhou.aliyuncs.com/acs/curl:8.1.2
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true
---

Run the following command to deploy the sleep application.

kubectl apply -f sleep.yaml

Step 2: Create a ServiceEntry and a DestinationRule

Because the LLM service is external to the mesh, you must create a ServiceEntry to register the service. This allows the mesh to manage it. For more information, see Create a service entry.

In this step, you create a ServiceEntry to register the Model Studio service with ASM. Use the following YAML content:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: dashscope
  namespace: default
spec:
  hosts:
    - dashscope.aliyuncs.com
  ports:
    - name: http-port
      number: 80
      protocol: HTTP
      targetPort: 443    # Used with a DestinationRule to upgrade HTTP to HTTPS
    - name: https-port
      number: 443
      protocol: HTTPS
  resolution: DNS

To enable the sidecar to upgrade HTTP requests on port 80 to HTTPS, create a corresponding DestinationRule with the following YAML content:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: dashscope
  namespace: default
spec:
  host: dashscope.aliyuncs.com
  trafficPolicy:
    portLevelSettings:
    - port:
        number: 80
      tls:
        mode: SIMPLE

Run the following commands in sequence to verify that the sidecar has upgraded the protocol from HTTP to HTTPS.

export API_KEY=${your_dashscope_api_key}  # Replace with your actual API key.

kubectl exec deploy/sleep -- curl -v 'http://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions' \
--header "Authorization: Bearer ${API_KEY}" \  
--header 'Content-Type: application/json' \
--header 'user: test' \
--data '{
    "model": "qwen-turbo",
    "messages": [
        {"role": "user", "content": "Who are you?"}
    ],
    "stream": false
}'

Expected output:

{"choices":[{"message":{"role":"assistant","content":"I am a large-scale language model from Alibaba Cloud. My name is Qwen."},"finish_reason":"stop","index":0,"logprobs":null}],"object":"chat.completion","usage":{"prompt_tokens":10,"completion_tokens":16,"total_tokens":26},"created":xxxxxxxx,"system_fingerprint":null,"model":"qwen-turbo","id":"xxxxxxxxxxxxxxxxxx"}

Step 3: Configure the LLMProxy plugin

Create a file named WasmPlugin.yaml with the following content.

apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: asm-llm-proxy
  namespace: default
spec:
  imagePullPolicy: Always
  phase: AUTHN
  selector:
    matchLabels:
      app: sleep
  url: registry-cn-hangzhou.ack.aliyuncs.com/test/asm-llm-proxy:v0.2
  pluginConfig:
    api_key: ${your_dashscope_api_key}
    deny_patterns:
    - .*账号.*     # Denies requests containing the Chinese word for "account" (账号).
    hosts:
    - dashscope.aliyuncs.com    # This plugin applies only to requests sent to dashscope.aliyuncs.com.
    intelligent_guard:   # Configure a private LLM service to check requests for sensitive information.
      # For demonstration purposes, this example still uses the Model Studio service for validation.
      api_key: ${your_dashscope_api_key}
      host: dashscope.aliyuncs.com
      model: qwen-turbo
      path: /compatible-mode/v1/chat/completions
      port: 80  # The HTTP port from the ServiceEntry.

The following table describes the parameters in the pluginConfig section.

The intelligent_guard uses a standard OpenAI interface to determine whether a request sent to the LLM contains sensitive information. If the private LLM determines that the request contains sensitive information, the request is denied and a reason is returned. For demonstration purposes, this example still calls the Model Studio service for this check.

Run the following command to create the WasmPlugin.

kubectl apply -f WasmPlugin.yaml

Testing

1. Run the following command to verify that a request without an API key successfully accesses the LLM service.

   kubectl exec deploy/sleep -- curl 'http://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions' \
   --header 'Content-Type: application/json' \
   --data '{
       "model": "qwen-turbo",
       "messages": [
           {"role": "user", "content": "Who are you?"}
       ],
       "stream": false
   }'

   Expected output:

   {"choices":[{"message":{"role":"assistant","content":"I am a large-scale language model from Alibaba Cloud. My name is Qwen."},"finish_reason":"stop","index":0,"logprobs":null}],"object":"chat.completion","usage":{"prompt_tokens":10,"completion_tokens":16,"total_tokens":26},"created":xxxxxxx,"system_fingerprint":null,"model":"qwen-turbo","id":"xxxxxxxxx"}

2. Run the following command to test that a request containing the sensitive word "account" will be rejected.

   kubectl exec deploy/sleep -- curl 'http://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions' \
   --header 'Content-Type: application/json' \
   --data '{
       "model": "qwen-turbo",
       "messages": [
           {"role": "user", "content": "I like red bean zongzi, and my QQ account is 1111111"}
       ],
       "stream": false
   }'

   Expected output:

   request was denied by asm llm proxy

3. Test a request that contains sensitive information but does not match any pattern in deny_patterns, and verify that the intelligent_guard identifies the sensitive information and blocks the request.

   kubectl exec deploy/sleep -- curl -s 'http://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions' \
   --header 'Content-Type: application/json' \
   --data '{
       "model": "qwen-turbo",
       "messages": [
           {"role": "user", "content": "Our company will hold a high-level internal meeting on September 10th. The topic is how to better serve customers. Please write an opening speech for me."}
       ],
       "stream": false
   }'

   external service returned deny: {"result": "deny", "reason": "The meeting content involves details and topics of a high-level internal company meeting, which is confidential information that cannot be disclosed."}

   As shown in the output, the LLM successfully identified that the request may contain sensitive information. The LLMProxy plugin then denied the request from being sent to the external LLM service. In a production environment, deploy the detection model privately to ensure no sensitive data is exposed.