Throttling

Features

• Prevent resource overload: Limit requests by consumer, header, query parameter, cookie, client IP, or model name to prevent overloads. Combine with a caching policy for better performance.

• Dynamic traffic control: Apply throttling per second, minute, hour, or day to maintain stability under high concurrency.

• Multiple matching rules: Supports exact match, prefix match, regex match, and any match for flexible rule targeting.

• Multiple throttling modes: Throttle by token consumption, request count, or concurrency.

• Fine-grained model control: Set different thresholds per model name to protect high-cost resources.

• Global limits: Apply an API-wide fallback limit on total token consumption, request count, and concurrency.

• Prevent malicious attacks: Throttle specific consumers, headers, query parameters, cookies, or client IPs to block crawlers and automated abuse.

Use cases

• High-concurrency scenarios: During e-commerce promotions, limit token consumption per time period to prevent malicious high-frequency calls and ensure stability and fairness.

• AI service calls: Throttle calls to your LLM APIs to prevent service degradation or system crashes caused by traffic bursts.

• Multi-tenant systems: Assign separate quotas per tenant to ensure fairness and resource isolation.

• Fine-grained model control: Set per-model limits (for example, different quotas for qwen-max and qwen-plus) to protect high-cost resources.

• Global traffic protection: Cap total token consumption, request count, and concurrency across the entire API.

• Malicious attack protection: Defend against crawlers, DDoS attacks, and API abuse.

Procedure

1. Go to the Instances page in the AI Gateway Console. In the top menu bar, select the region where your target instance is located, and then click the target instance ID.

2. In the left-side navigation pane, click Model API, and then click the target API Name to go to the API Details page.

3. Click Policies and Plug-ins, then turn on the Throttling switch and configure the related parameters.

Note A request can match a maximum of 10 rules simultaneously.

Matching rules

The conditions By consumer, By request header, By request query, and By request cookie support the following four matching rules. Priority: exact match > prefix match > regex match > any match.

Note If you configure multiple rules, a request is throttled if it matches any of them. The By client IP and By model conditions use fixed matching methods and do not require manual rule selection.

Limit units and modes

For each throttling rule, select a limit unit that corresponds to a different metering method:

The throttling policy supports the following time windows, used in combination with a limit unit:

Note Concurrency throttling does not require a time window. You can set the maximum concurrency directly.

Configuration examples

FAQ

Q: What is the maximum number of throttling rules I can configure?

A: A request can match up to 10 rules simultaneously. Combine rules across dimensions as needed, but fewer rules yield better performance.

Q: How do multiple rules interact with each other?

A: Rules use OR logic—throttling triggers when any rule matches. Rules with the same condition and match key merge into a single rule group.

Q: Can I use a global limit and standard throttling rules at the same time?

A: Yes. The global limit is a fallback that applies to the entire API without distinguishing keys, while standard rules provide fine-grained control per dimension. Throttling triggers if any limit is exceeded.

Q: What is the difference between throttling by model and by request header?

A: Throttling by model automatically converts to an exact match on the x-higress-llm-model header. For prefix or regex matching, use "By request header" with parameter name x-higress-llm-model.

Q: Can I combine the token, request count, and concurrency limit units?

A: Yes. Add multiple rules for the same condition with different limit units (for example, a per-minute token limit and a max concurrency limit for one model). They count independently, and throttling triggers if any limit is reached.

Q: Why are there no options for matching rule and time window when throttling by client IP?

A: Client IP throttling is simplified—enter an IP address or range and the system handles matching automatically. To apply a limit to all client IPs, enter 0.0.0.0/0.

Q: Does the order of the rules affect throttling behavior?

A: Rules support drag-and-drop sorting. However, since rules use OR logic (throttling triggers if any rule matches), order does not affect behavior.

Q: How long does it take for updated throttling configurations to take effect?

A: Changes typically take effect within seconds after the system pushes the new rules to the gateway data plane.

Q: How accurate is throttling in a distributed architecture?

A: In distributed systems, throttling counts may have slight deviations. The actual allowed requests may differ from the configured limit depending on request volume, rate, and backend latency.