Enable AI Guardrails in AI Gateway

Prerequisites

Workflow overview

The workflow consists of two steps: create a protection configuration in the AI Guardrails console, and then associate the configuration with the target instance in the AI Gateway console.

Step 1: Create a configuration in the AI Guardrails console

Step 2: Associate the configuration in the AI Gateway console

1. Log on to the AI Gateway console, go to the Instances page, select the region where the target instance is located from the top menu bar, and click the target instance.

   Important

   • AI Gateway automatically selects the nearest AI Guardrails region based on its own region. The mapping is shown in the following table. AI Guardrails in US (Virginia) does not support AI Gateway integration, so AI Gateway nodes in the United States are routed to Germany (Frankfurt).

     Gateway console region	Guardrails console region
     Asia Pacific - China	China (Shanghai)
     Asia Pacific - Others	Singapore
     China (Hong Kong)	China (Hong Kong)
     Europe and Americas	Germany (Frankfurt)
     Other regions	China (Shanghai)

2. Find the instance that you want to configure and make sure that its engine version is 2.1.26 or later. If the version is earlier than 2.1.26, manually upgrade the instance on the instance page first.

3. In the left-side navigation pane, click Model API, and then click the target API name to go to the API details page.

4. Go to the API Details page and click the AI Guardrails section to edit it. If SLR authorization is not complete, follow the page prompts to complete the authorization and allow AI Gateway to access AI Guardrails.

5. Select the configuration that you created in the AI Guardrails console in the previous step and associate it with the current gateway instance. If you are switching from an earlier configuration, manually switch to the new version.

6. The configuration takes effect immediately after you save it. No restart or redeployment is required.

Review after delivery details

data: {"id":"chatcmpl-xxxx","object":"chat.completion.chunk","created":1793104383,"model":"qwen-max","choices":[{"index":0,"delta":{"content":"The content is not compliant. The response has been recalled."},"finish_reason":"content_filter","x_guardrail":{"action":"recall","scope":"all","replace_text":"The content is not compliant. The response has been recalled."}}]}

data: [DONE]
          

// In the SSE streaming loop
for await (const chunk of stream) {
  const choice = chunk.choices?.[0];

  // Check for a recall event
  if (choice?.finish_reason === 'content_filter'
      && choice?.delta?.x_guardrail?.action === 'recall') {
    // Clear the displayed content
    clearDisplayedContent();
    // Display the replacement text
    displayText(choice.delta.x_guardrail.replace_text);
    break;
  }

  // Display the streamed content normally
  appendText(choice?.delta?.content);
}
          

Cautions

• Configuration scope: One guardrail configuration can be associated with multiple gateway instances. After you modify the guardrail configuration, the changes take effect on all associated instances.

• Switching the review mode: To switch between review before delivery and review after delivery, change the review mode in the AI Guardrails console. No change is required on the gateway side.

• Custom detection agent latency: After you enable a custom detection agent, the check latency increases from about 200 ms to 200 ms to 2 s, depending on the model that you select. Choose the agent based on your latency requirements.

• Recall events require client support: In review after delivery mode, recall actions require client-side handling. If the client does not implement recall logic, non-compliant content remains on the user interface.

• AI Gateway engine version requirement: The engine version of the AI Gateway instance must be 2.1.26 or later. Otherwise, you cannot associate a guardrail configuration.

• Migrating from an earlier version: If you are using an engine version earlier than 2.1.26 with guardrails enabled, manually upgrade the engine version first, and then perform the preceding two steps to complete the configuration. You must also manually switch to the new version on the guardrail configuration page in AI Gateway. Version switching cannot be reverted, so proceed with caution. We recommend that you add an instance, complete and verify the configuration, and then switch traffic.