TokenWorks is Alibaba Cloud PAI's enterprise-grade, high-SLO inference platform for the agentic era. TokenWorks delivers LLM intelligent routing, cache-aware scheduling, KV cache storage, model warm-up, and global compute orchestration. Enterprises can run private large model inference at lower cost with stronger SLO guarantees and full operational control.
Why TokenWorks
Agent and AI coding workloads demand large model inference systems that handle high concurrency, long context windows, heavy cache dependency, and cost pressure. Traditional QPS throttling, stateless load balancing, and GPU-stacking scale-out cannot solve problems like low KV cache hit rates, TTFT jitter, GPU memory waste, and opaque multi-tenant cost attribution. Enterprises need an inference gateway and scheduling engine purpose-built for token traffic — one that maximizes compute efficiency.
Key capabilities
LLM intelligent routing: the dedicated traffic gateway for the token era. Natively compatible with OpenAI, Anthropic Messages API, Alibaba Cloud Bailian, and other protocols — clients such as Claude Code and Codex connect without code changes. Supports prefill-decode (PD) separation traffic orchestration, token-level rate limiting, model routing, weighted and prefix-based traffic splitting, multi-tenant API keys, and usage metering. This establishes cost governance for your enterprise AI platform.
Deep engine optimization: built for next-generation model architectures. The inference engine is fine-tuned for MoE and hybrid attention models. It natively supports large-scale expert parallelism (EP) and heterogeneous prefill-decode (PD) separation. The engine also integrates HiSparse memory sparsity, DSpark/DFlash block diffusion speculative decoding, PPD and Decode Radix Cache reuse optimizations, and PPU-side network communication and custom operator optimizations — delivering production-ready, deterministic performance gains.
Cache-aware scheduling: route traffic to where the cache lives. The four-level scheduling system — session affinity, prefix matching, KV cache reuse, and load balancing — turns KV cache from a black box into a manageable asset. Cache hit rates increase, TTFT drops, and hot-instance overload is prevented.
KV Cache Store: offload GPU memory pressure to distributed storage. Frequently reused KV cache is offloaded from GPU memory to CPU memory, shared disk, or a dedicated cache pool, expanding prefix storage capacity by tens of times. KV layout and compression are optimized for sparse attention structures such as GLM and DeepSeek, maintaining high hit rates and low latency even in long-context workloads.
Model warm-up and memory caching: cold-start in seconds. Image and weight pre-warming for large model deployments compresses model loading time from minutes to seconds, even for models hundreds of gigabytes in size. Combined with spot instance scheduling, this makes elastic scaling and automatic fault recovery operationally practical.
Global scheduling: pool compute across regions. When inference resources are scarce in the current region, traffic is automatically routed to idle inference instances in other regions. This achieves cross-region compute pooling and reuse while maintaining business continuity and cost efficiency.
Key concepts
TokenWorks comprises the following functional modules:
|
Module |
Description |
|
Workspace |
The top-level management unit in TokenWorks. Creating a workspace generates a unified entry point (base URL and API key). All inference services and KV cache services belong to the workspace. |
|
Inference service |
A model inference instance within the workspace. Create one from ModelGallery or from a custom model. Each inference service retains its own entry point while sharing the workspace's unified entry point. |
|
KV Cache |
The cache service within the workspace. A multi-tier caching and lookup mechanism spanning GPU memory, CPU memory, local SSD, and remote storage improves response efficiency for long-context and high-frequency requests. |
|
Intelligent Routing |
The traffic scheduling component behind the workspace's unified entry point. Routes requests to the appropriate inference service instances based on your configured scheduling policy. |
|
Configuration Center |
The management entry point for advanced scheduling configuration, including scheduling policies (session and cache affinity), traffic splitting, rate limiting, traffic mirroring, and global scheduling. |
|
User Management |
Manages users within the workspace. Supports adding, editing, removing, enabling, and disabling users; provides API key management; and assigns permissions by role (regular user, administrator, and system administrator). |
|
Real-time monitoring |
Displays near-real-time performance metrics — throughput, TTFT/TPOT, cache hit rate, and QPS — and the operational status of each component. Use it to quickly identify anomalies and performance bottlenecks. |