AI cache-Microservices Engine(MSE)-阿里云帮助中心

Repeated or similar LLM queries across users drive up token costs and add unnecessary latency. The ai-cache plug-in caches LLM responses at the gateway level so that identical requests return instantly from Redis instead of calling the upstream model again. It supports both streaming and non-streaming responses and works with the OpenAI-compatible protocol out of the box.

How it works

A request arrives at the gateway. The plug-in extracts a cache key from the request body using a GJSON PATH expression. By default, the expression messages.@reverse.0.content extracts the content of the most recent message.
The plug-in looks up the key in Redis.
- Cache hit -- Returns the cached response directly without calling the upstream LLM.
- Cache miss -- Forwards the request to the upstream LLM, then stores the response in Redis for future use.

Running attributes

Attribute	Value
Execution stage	Authentication stage
Execution priority	10

Configure the plug-in

Basic setup

The only required parameter is redis.serviceName. This minimal configuration caches LLM responses in Redis with default settings:

redis:
  serviceName: my-redis.dns
  timeout: 2000

With this configuration, the plug-in:

Extracts the cache key from the last message content (messages.@reverse.0.content)
Stores cached entries under the higress-ai-cache key prefix
Keeps entries indefinitely (TTL of 0)

Set a TTL and isolate cache entries

Use cacheTTL to expire stale entries and cacheKeyPrefix to isolate cache entries across environments, tenants, or applications:

cacheKeyPrefix: "my-app-cache"
cacheTTL: 3600
redis:
  serviceName: redis.my-ns.svc.cluster.local
  servicePort: 6379
  timeout: 2000
  username: <your-redis-username>
  password: <your-redis-password>

Replace the following placeholders with your actual values:

Placeholder	Description	Example
`<your-redis-username>`	Redis authentication username	`cache_user`
`<your-redis-password>`	Redis authentication password	`P@ssw0rd`

Common isolation patterns for cacheKeyPrefix:

Pattern	Prefix example	Use case
Per-environment	`prod-ai-cache`, `staging-ai-cache`	Prevent staging data from polluting production cache
Per-tenant	`tenant-a-cache`, `tenant-b-cache`	Isolate cache entries in a multi-tenant gateway
Per-application	`chatbot-cache`, `summarizer-cache`	Separate cache pools for different LLM applications

Advanced cache key extraction

The plug-in uses GJSON PATH expressions to extract cache keys from request bodies. The default expression messages.@reverse.0.content reverses the messages array and returns the content of the first element, which is typically the most recent user message.

The following examples show common extraction patterns.

Filter by role

To use only the content from the last message where role is "user":

messages.@reverse.#(role=="user").content

Combine multiple messages into a single key

To concatenate the content of all user-role messages into an array:

messages.@reverse.#(role=="user")#.content

Select a specific message by index

The pipe syntax lets you pick a specific element. This expression selects the second user message as the cache key:

messages.@reverse.#(role=="user")#.content|1

For the full GJSON PATH syntax reference, see the official documentation. Test expressions interactively in the GJSON Playground.

Configuration reference

Name	Data type	Required	Default value	Description
cacheKeyFrom.requestBody	string	No	`"messages.@reverse.0.content"`	GJSON PATH expression used to extract the cache key from the request body. By default, reverses the `messages` array and takes the `content` of the first element.
cacheValueFrom.responseBody	string	No	`"choices.0.message.content"`	GJSON PATH expression used to extract the cached value from a non-streaming response body.
cacheStreamValueFrom.responseBody	string	No	`"choices.0.delta.content"`	GJSON PATH expression used to extract the cached value from a streaming response body.
cacheKeyPrefix	string	No	`"higress-ai-cache"`	Prefix for all Redis cache keys. Use different prefixes to isolate cache entries across environments or tenants.
cacheTTL	integer	No	0	Cache entry expiration in seconds. `0` means the cache never expires.
redis.serviceName	string	Yes	-	Fully qualified domain name (FQDN) of the Redis service. Example: `my-redis.dns` or `redis.my-ns.svc.cluster.local`.
redis.servicePort	integer	No	6379	Port of the Redis service.
redis.timeout	integer	No	1000	Redis request timeout in milliseconds.
redis.username	string	No	-	Username for Redis authentication.
redis.password	string	No	-	Password for Redis authentication.
returnResponseTemplate	string	No	See below.	Response template for non-streaming cache hits. `%s` is replaced by the cached value.
returnStreamResponseTemplate	string	No	See below.	Response template for streaming cache hits. `%s` is replaced by the cached value.

Default response templates

Non-streaming (returnResponseTemplate):

{"id":"from-cache","choices":[%s],"model":"gpt-4o","object":"chat.completion","usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

Streaming (returnStreamResponseTemplate):

data:{"id":"from-cache","choices":[{"index":0,"delta":{"role":"assistant","content":"%s"},"finish_reason":"stop"}],"model":"gpt-4o","object":"chat.completion","usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

data:[DONE]