Create and manage a KV cache service

Key benefits

Local prefix caching in inference engines such as vLLM and SGLang is limited to a single replica and constrained by GPU memory. The KV cache service provides cluster-level capabilities:

• Cluster-wide sharing: All inference service replicas bound to the same KV cache service share a single cache. Regardless of which replica handles a request, it can access cached results computed by any other replica. The cache hit ratio does not drop when you scale out.

• Significantly lower TTFT: Requests that hit the cache skip the entire prefill phase for the shared prefix, so TTFT drops to roughly the time needed to process only the new input in the current turn. The longer the prefix — such as thousands of tokens of conversation history or retrieved documents — the more computation is skipped and the greater the TTFT reduction.

• Higher throughput, lower cost: Skipping prefill frees GPU compute, so the same resources handle more concurrent requests. The average compute cost per request drops as the hit rate increases.

• Beyond GPU memory limits: The KV cache service uses tiered persistence across memory, disk, and remote storage. Hot prefixes stay in memory for low latency, while cold data is automatically moved to disk or remote storage to extend capacity. You can cache far more prefix data than GPU memory alone allows, maintaining a higher hit rate. Storage capacity isn't limited to a single machine — add nodes to scale linearly with minimal reconfiguration.

• High availability and automatic fault recovery: After a node failure, metadata is cleaned up automatically and cache state is restored. Disk-based cache data survives process restarts and remains reusable, so inference services are unaffected. Scaling operations minimize data loss and maintain service stability.

How it works

The KV cache service sits between the inference engine and the underlying storage. The core workflow has two steps:

• Store after compute: After the inference engine completes the prefix prefill, it writes the generated KV tensors to the KV cache service, which persists them through the tiered storage policy (memory → disk → remote storage).

• Fetch before compute: When a subsequent request arrives, the system checks the KV cache service first for a matching prefix. On a cache hit, the cached KV tensors are returned directly and the prefill for that prefix is skipped.

Use cases

The KV cache service delivers the greatest benefit when different requests share large amounts of prefix content:

• Multi-turn conversation: Conversation history accumulates across turns. Cache hits mean each turn computes KV only for the new message, and TTFT decreases with each turn.

• RAG (retrieval augmented generation): Retrieved documents appear repeatedly across different requests. With cache hits, each request only computes the KV for the user query.

• Shared system prompt: Multiple users share the same long system prompt. With cache hits, each request only computes the KV for the user input portion.

• Multi-model or multi-instance cluster: Multiple inference services bound to the same KV cache service share the cache. Hot prefixes are computed only once, improving overall resource utilization.

• Elastically scaled inference services: New replicas after scaling hit the existing cache immediately — no warm-up needed. Scaling doesn't lose cached data, so traffic spikes and drops are handled smoothly.

The KV cache service doesn't provide inference capabilities on its own. Use it with an inference service. When you deploy the inference service, select a KV cache service — the cache is applied automatically after binding — no inference engine configuration is required.

Create a KV cache service

1. In the left-side navigation pane, click Service and then click the KV Cache tab.

2. Click Create a cache service and configure the following parameters:

   • Service Name: The name can contain only lowercase letters, digits, and underscores. The name must start with a lowercase letter.

   • Resource allocation: The KV cache service uses only CPU and memory — no GPU is required.

     • Resource Type: Select EAS Resource Group or Resource Quota.

     • Resource allocation: Set the number of replicas, CPU, memory (GB), and system disk (GB).

       Cache capacity depends primarily on memory size. Increase memory based on the expected amount of cached data.

   • Redis configuration: We recommend an instance specification of at least 16 GB. Use a separate Redis instance from the one that the workspace uses to store user and usage data. This prevents data interference.

     Choose a configuration method based on whether you already have a Redis instance:

     • Create a new instance: Recommended when you need to create a new instance. The system automatically creates a Tair instance and configures the allowlist.

       After you click Create Redis instance, the Tair instance creation progress appears. Do not refresh the page during this process. If you refresh the page, go to the Tair console to complete the remaining steps. After creation, the Redis address, Redis username (default: default), and Redis password are displayed. Store this information securely.

     • Select an existing instance: If you have an existing Tair or Redis instance in your account, select it to avoid creating a duplicate instance and incurring additional costs.

     • Manual input: The Redis instance and the workstation must be in the same VPC; otherwise, the connection will fail.

3. Review the configuration, and then click Deployment.

Manage a KV cache service

• View service monitoring: Click the name of a KV cache service to go to its details page, and then click the Monitoring tab. See FlashKV monitoring metrics to review overall service performance or troubleshoot issues.

• Delete a KV cache service: On the KV Cache page, find the cache service that you want to delete, and then click Delete in the Actions column.

  • After you delete the KV cache service, inference services in the workspace stop using the cache. Cached data can't be recovered, but the inference services themselves are unaffected and continue to accept requests.

  • Deleting a cache service doesn't automatically delete the Redis instance. Release the Redis instance manually as needed.

FlashKV monitoring metrics

FlashKV provides service-level and instance-level monitoring dashboards. The chart layout and metric definitions are essentially the same across both dashboards. The difference is the aggregation scope:

• The service-level dashboard aggregates data across all instances, useful for observing overall trends.

• The instance-level dashboard shows per-instance data, for detailed performance analysis and troubleshooting.

The following table lists all available metrics: