TokenWorks configuration center-Platform For AI(PAI)-阿里云帮助中心

Scheduling policy

The scheduling policy is configured across three components:

Main scheduling policy: The core routing pipeline. Routes requests to inference instances through three priority layers — session affinity → prefix cache → load balancing — matching each layer in order and routing on the first hit.
Prefix index: A prefix trie that persists inside the scheduler, tracking the prompt prefix cache state across all instances to estimate the theoretical cache hit rate ceiling. When prefix cache scheduling is enabled, the index also serves as the routing match source. Configure its memory bounds and cleanup interval here.
PD disaggregation: Applies only in Prefill-Decode (PD) disaggregated deployment mode. Configures a separate scheduling policy for the Decode phase.

Main scheduling policy

The main scheduling policy runs in three priority layers. Each request passes through the layers in order. If a layer matches, the request is routed immediately. Otherwise, it falls through to the next layer. Session affinity and prefix cache are optional and can be toggled independently. Load balancing is required.

Note

The scope of the main scheduling policy depends on the deployment mode. In PD integrated mode, it acts as the global routing pipeline for all requests. In PD disaggregated mode, it applies only to the Prefill role; the Decode phase uses the separate policy configured in the PD disaggregation section below.

Layer 1: Session affinity

Routes requests from the same session to the same inference instance, preserving context across turns. Suited for multi-turn conversations and stateful inference.

When enabled, requests must include the configured session header to trigger affinity routing. Requests without the header fall through to the prefix cache layer.

Parameter	Description
Affinity mode	Controls how strictly requests are pinned to their bound instance. Soft: Prefers the bound instance but falls back to standard scheduling when the instance load exceeds average load × soft max load ratio. Use when performance matters but occasional instance switching is acceptable. Strict: Always routes to the bound instance, ensuring strong context continuity at the cost of potential load imbalance.
Session header	The HTTP request header field used to identify the session ID. Defaults to `X-Session-Id`. Must match the header name sent by the client — if the request doesn't include this header, affinity routing doesn't activate. Note Clients known to send a session header automatically: • Claude Code: X-Claude-Code-Session-Id (one per session) Other popular LLM clients (OpenAI/Anthropic SDK, LangChain, and others) don't send any session header by default. The caller must explicitly inject the header into each request. Requests without the header are handled by the standard scheduling policy.
Session TTL	How long a session binding is retained, in seconds. The binding is cleared automatically when it expires. Default: 1800 seconds.
Max entries	The maximum number of session records the affinity cache holds. When the limit is reached, the oldest records are evicted. Default: 100000.
Soft max load ratio	Applies in Soft mode only. When the instance load exceeds average load × soft max load ratio, affinity routing is abandoned and the request is sent to a less-loaded instance. Default: 1.5.

Layer 2: Prefix cache

Routes requests to instances with a matching prompt prefix cache. This reduces redundant computation and improves cache hit rates. Best suited for workloads with highly repetitive input prefixes, such as fixed system prompts or batch document processing. Requests that don't match fall through to the load balancing layer.

Parameter	Description
Load tolerance factor	When a cache-hit instance's load exceeds average load × (1 + tolerance factor), routing no longer prefers that instance. Default: 0.3.
Waiting request threshold	When the number of pending requests on an instance exceeds this value, the instance is considered overloaded. Even with a cache hit, requests are no longer routed to it, preventing single-instance overload. Default: 2.
Hit rate threshold	If the prefix match length is less than this fraction of the total prompt length, the result is treated as a cache miss. Default: 0.50.

Layer 3: Load balancing

When neither session affinity nor prefix cache produces a match, the load balancing algorithm distributes the request across inference instances. The following algorithms are available:

Algorithm	Best for
`least-token`	Routes to the instance processing the fewest tokens. Best for workloads with large variation in token length, distributing compute pressure evenly.
`least-request`	Routes to the instance with the fewest active requests. Best for workloads where request processing time is roughly uniform.
`round-robin`	Distributes requests across instances in rotation. Best for homogeneous instances with stable, predictable load.

Prefix index

The prefix index is a trie that persists inside the scheduler. It tracks the prefix cache state across all instances and computes the theoretical cache hit rate ceiling. When prefix cache scheduling is enabled, the index also drives routing decisions, directing requests to the instance with the highest estimated cache hit rate. The following parameters control the index's memory bounds and cleanup interval.

Parameter	Description
Cache TTL	How long a prefix index entry is retained, in seconds. Expired entries are cleared automatically. Default: 259200 seconds (3 days). If sessions typically complete within one hour and GPU memory is constrained, reduce this to 3600 seconds to reclaim cold data faster.
Maintenance interval	How often the background task scans and removes expired index entries, in seconds. Default: 60 seconds, which suits most workloads. Increase this value only when the cluster CPU load is consistently high.
Memory threshold ratio	Together with the reserved memory setting, this determines the memory eviction trigger point for the scheduler process. Whichever limit is hit first takes effect. When the scheduler process memory exceeds container memory × this ratio, LRU (Least Recently Used) eviction removes the oldest index entries. Set to 0 to use the default value of 0.70. Decrease this value if the scheduler container experiences frequent OOM events. Increase it if index entries are evicted too aggressively, degrading routing decisions.
Reserved memory	The minimum amount of memory reserved for the container OS and runtime (in GiB). Together with the memory threshold ratio, this setting determines the memory eviction trigger point. When the scheduler process memory exceeds container memory − reserved memory, LRU eviction removes the oldest index entries. Set to 0 to use the default value of 1.50 GiB. Increase this value if the scheduler container is being killed by the system OOM killer. Decrease it if index entries are evicted too aggressively.

PD disaggregation

In PD disaggregated mode, Prefill and Decode run on separate instances. Configure an independent scheduling policy for Decode here.

Parameter	Description
PD disaggregation mode	The PD disaggregation implementation. Must match the inference engine used during deployment — for example, `sglang-mooncake`.
Decode scheduling policy	The independent scheduling policy for the Decode role when PD disaggregation is enabled. The Prefill role continues to use the main scheduling policy. Supports the same algorithms as the main scheduling policy's load balancing layer: `least-token`, `least-request`, `round-robin`.

Model routing

Routes each request to the corresponding inference service instance in the workstation based on the model field. When enabled, the Default model setting controls what happens when no matching instance is found.

Default model	Description
Strict match (no default configured)	Routes the request only when the `model` field exactly matches an inference service name in the workstation. Returns an error if no match is found.
Specified model	When the `model` field in the request doesn't match any inference service in the workstation, routes the request to the specified fallback model instance.

Split proxy

Forwards requests received by the workstation to external upstream API services — such as PAI Token Service — according to configured routing rules. This lets the workstation coordinate traffic between local inference services and external services, improving overall service stability.

When enabled, select a split strategy and configure one or more proxy entries in the proxy list.

Split strategy

Strategy	Description
Weight	Distributes requests across proxies according to their configured weight ratios. Best for gradual rollouts, A/B testing, or any scenario requiring proportional traffic control.
Prefix	Routes traffic to a proxy based on the prefix of the `model` field in the request.

Proxy list

The proxy list must contain at least one Local service entry, which represents the workstation's own inference service. Additional external upstream services can be added as extra proxies. Column descriptions:

Column	Description
Status	Whether this proxy entry is enabled. Disabled proxies don't receive forwarded requests.
Proxy address	The endpoint of the upstream API service. The Local service entry is fixed and represents the workstation's own inference service.
Model	The model name that replaces the `model` field in outgoing requests to this upstream. Leave blank for Local service.
API key	The API key for authenticating with this proxy. Leave blank for Local service.
Weight	The traffic weight assigned to this proxy under the Weight strategy. Proxies receive requests proportional to their weight. Displays Not enabled under the Prefix strategy.
Prefix	The prefix matching pattern for the request's `model` field. A trailing asterisk () acts as a wildcard. When multiple entries match, the longest prefix wins. Use a single asterisk () as a catch-all fallback.
Failover	When set to Participate, the system switches to this proxy (in list order) if other proxies are unavailable or fail. Local service doesn't support failover.

Note

External proxy services follow their own rate-limiting rules and aren't subject to the workstation's request throttling configuration.

Request throttling

Limits concurrent requests to the workstation to prevent backend inference instances from becoming overloaded. When the limit is exceeded, requests can be rejected immediately or queued. Set concurrency limits separately for the full request lifecycle, the Prefill phase, and the Decode phase.

Parameter	Description
Overflow handling	What happens when the concurrency limit is exceeded: Reject immediately (return an error right away) or Queue (place the request in a waiting queue).
Throttling scope	The level at which concurrency is counted: Instance: Checks each instance individually, routing only to instances whose load hasn't exceeded the limit. Service: Checks the entire cluster. Throttling triggers when the combined load of all instances exceeds the threshold (per-instance limit × instance count).
Total concurrency limit	A slot is held from request dispatch until the response completes, counting the full request lifecycle. Applies only in PD integrated deployment. Not counted in PD disaggregated deployment.
Prefill concurrency limit	Observation window: from request dispatch until the first token is received. This window covers both engine queuing and prefill computation — the two can't be distinguished, so both are counted as prefill-phase occupancy. Applies to the Prefill phase in PD integrated mode and to Prefill nodes in PD disaggregated mode.
Decode concurrency limit	Observation window: from the first token received until the response completes. Applies only in PD disaggregated deployment, limiting the Decode nodes. Not counted in PD integrated deployment.

Traffic mirroring

Copies live production requests at a configured sampling ratio and sends them to a target address. The target service processes mirrored traffic independently. This has no effect on the responses returned to real clients. Use for canary validation of new versions, performance comparisons, and data collection.

Parameter	Description
Target address	The endpoint that receives mirrored traffic.
Target model	The model name substituted in mirrored requests. Leave blank to keep the original request's model name.
Sampling ratio	The percentage of total traffic to mirror (0%–100%).
Auth token	The authentication token for accessing the target address.
Timeout	The timeout for each mirrored request, in milliseconds. After the timeout elapses, the mirror request is dropped silently — normal request processing is not affected.
Log requests	Whether to record detailed logs for mirrored requests, for later analysis and comparison.

Global scheduling

When inference resources in the current region are insufficient, global scheduling routes traffic to idle inference instances in other regions. This enables cross-region compute pooling and efficient resource reuse.

After enabling global scheduling, add TokenWorks services from other regions to the member list:

Member name: A custom display name for the member.
URL / Token: The base URL and API key of the TokenWorks service deployed in the other region.