The load-aware routing component schedules traffic to backend services based on their real-time load. It periodically collects backend metrics such as request queue length, KV Cache utilization, and running request count, scores each backend based on configured metric weights, and routes requests preferentially to backends with lower load. This improves resource utilization and reduces inference latency.
The load-aware routing component can only be used with the Weighted Round Robin (WRR) scheduling algorithm, and cannot be added to the same service extension as other components.
Function type
Route scheduling
Field description
ComponentConfig
Parameter | Type | Required | Default Value | Description |
Config | Yes | - | The load-aware routing configuration. |
LoadAwareRoutingConfig
Parameter | Type | Required | Default Value | Description |
MetricCollector | Yes | - | The metric collection configuration. | |
Scheduler | Yes | - | The route scheduling configuration. |
MetricCollectorConfig
Parameter | Type | Required | Default Value | Description |
Timeout | Integer | Yes | - | The timeout period for metric collection responses. Unit: seconds. Valid values: |
Interval | Integer | Yes | - | The interval for metric collection. Unit: seconds. Valid values: |
HttpMetricCollect | Yes | - | The HTTP metric collection configuration. |
HttpMetricCollectorConfig
Parameter | Type | Required | Default Value | Description |
Host | String | No | - | The Host header value for metric collection requests. Must be a valid domain name. If not specified, the backend server address is used by default. |
Path | String | Yes | - | The request path for metric collection. Must start with a forward slash ( |
Method | String | No |
| The HTTP method for metric collection requests. Valid values: |
SchedulerConfig
Parameter | Type | Required | Default Value | Description |
Scorers | Yes | - | The list of route scheduling scorer configurations. At least one scorer must be configured. The system sorts backends based on the scores of each metric and routes requests preferentially to backends with lower load. |
ScorerConfig
Parameter | Type | Required | Default Value | Description |
Name | Yes | - | The name of the metric used for scoring. | |
Weight | Integer | Yes | - | The weighted proportion of this metric after linear normalization. Valid values: |
ScorerNameEnum
Value | Description |
| The total number of queued requests. This is the number of requests currently queued and waiting to be processed. |
| The KV Cache utilization percentage. This is the current utilization of the KV Cache used to cache intermediate inference results. |
| The number of running requests. This is the number of requests currently being processed. |
Error codes
HTTP Status Code | Error Message | Cause |
503 |
| All backends have reached saturation and no backend is available to handle traffic. |