AI Gateway (beta)

Features

AI Gateway proxies requests between your applications and AI service providers with these core capabilities:

• Unified access: Connect to multiple AI service providers through a single endpoint. Compatible with the OpenAI API standard, so you can switch models without changing code.

• Security: Built-in rate limiting, IP allowlist, and API key authentication protect your AI services from abuse and attacks.

• High availability: Automatic failover and request retry ensure business continuity. When a primary provider is unavailable, requests switch to a backup provider.

• Observability: Monitor request counts, Token usage, response times, and costs in real time, with detailed request logs and audit trails.

How it works

AI Gateway runs on ESA's global edge nodes. Requests are processed as follows:

1. Request reception: The client sends an API request to the unified endpoint, routed to the nearest edge node.

2. Security verification: The edge node authenticates the request using API keys, IP allowlists, or other configured methods.

3. Routing and forwarding: The request is forwarded to the target AI service provider (such as Alibaba Cloud Model Studio or OpenAI) based on configured routing rules.

4. Result caching: The edge node caches the upstream response and returns it to the client.

5. Monitoring records: Records Sampled Logs and metrics such as Requests, Token Usage, and Error Count.

Routing mechanism: Specify the target AI model with the model parameter in {provider}/{model-name} format. Examples:

• openai/gpt-4o: Calls OpenAI's GPT-4o model.

• deepseek/deepseek-chat: Calls DeepSeek's chat model.

• qwen/qwen-plus: Calls Alibaba Cloud Model Studio's Qwen-Plus model.

Use cases

import OpenAI from "openai";

const client = new OpenAI({
  defaultHeaders: {
    "esa_aigw_auth": `Bearer ESA_API_TOKEN`,
  },
  baseURL: "https://YOUR_DOMAIN/v1",
});

const response = await client.chat.completions.create({
  model: "qwen/qwen-plus",  // Specify the model to use.
  messages: [{ role: "user", content: "What is Edge Security Acceleration?" }],
});

import OpenAI from "openai";

const client = new OpenAI({
  defaultHeaders: {
    "esa_aigw_auth": `Bearer ESA_API_TOKEN`,
    "x-ai-route-metadata": "YOUR_IFELSE_CONDITION_RULE",  // Required when an If...Else condition is configured.
  },
  baseURL: "https://api.example.com/v1",
});

const response = await client.chat.completions.create({
  model: "dynamic/ai-test",  // Routing configuration
  messages: [
    { role: "user", content: "Hello!" }
  ],
});

console.log(response.choices[0].message.content);

Notes

• API Key Management: AI Gateway requires API keys from upstream AI service providers. Store these keys securely and rotate them regularly. The keys are encrypted and stored in ESA.

• Model compatibility: The unified interface follows the OpenAI API standard. Some provider-specific features may not be supported.

• Monitoring data latency: Metrics may have a 1- to 5-minute delay. For near-real-time needs, supplement with log analysis.

Steps

Routing configuration

Configure routing rules to match models based on request characteristics. Add failover rules to switch to a backup provider or alternate model when a service becomes unavailable.

1. In the AI Gateway application, select the Routing Configuration tab, click Add Route, and then enter a custom Name on the page that appears.

2. After you create the routing configuration, click Edit in the Actions column for the corresponding route.

3. On the edit page, configure model routing visually:

   1. Configure routing conditions: In the visual model, click the add icon between the Start and Model Service flows, then click Add Condition to generate an If...Else decision flow. Next, click Configure in the If...Else flow and then click Add Condition in the panel on the right.

      • metadata.: Enter the key to be evaluated, such as test-key.

      • Logical operator: Select a logical operator, such as == (equals) or != (not equals).

      • Value: Enter the value that corresponds to metadata., such as ok.

      Note

      By default, AI Gateway uses the primary provider when the condition is true and the backup provider when it is false. Edit the model configuration to change this behavior.

   2. Configure the model service: For the true and false branches, click the add icon and select Add Model Service (or Add Condition). Then, click Configure on the Model Service process block to add model information:

      • AI Service Provider: Select an AI service provider from the drop-down list, such as Qwen.

      • Model Name: Select a model from the drop-down list, such as qwen3-max.

      • API Key: Select an added API key, such as qwen-api-key.

      Click Add Fallback Model to add a backup model for failover.

4. After completing the configuration, click Save and Deploy. The system uses the x-ai-route-metadata header to match routing conditions and select a model service.

Analytics and logs

ESA collects key data from all AI Gateway requests to help you analyze your AI application performance.

1. In the AI Gateway application, select the Analytics tab.

2. Select a time range to view these metrics:

   • Requests: Total requests through this AI Gateway in the selected time range.

   • Token Usage: Estimated tokens used by this AI Gateway in the selected time range.

   • Error Count: Error requests recorded for this AI Gateway in the selected time range.

   • Sampled Logs: Detailed request and response information, including Edge Status Code, Time, Model, Request, and Response.

Supported AI service providers

AI Gateway supports the following AI service providers. More providers will be added.

Billing

AI Gateway defaults to a Free Version at no additional cost, ideal for individual developers and personal projects. A paid plan for enterprise users will be available later.