Cache

Semantic cache concepts

Benefits

• Dual-mode cache system: Provides the flexibility to choose between two cache modes and dynamically adjust the matching threshold based on your business needs.

  1. Precise cache: Based on the Redis key-value storage architecture, this mode provides millisecond-level responses to identical requests.

  2. Semantic cache: Uses the DashVector vector database (part of Alibaba Cloud Vector Retrieval Service) to intelligently match semantically similar requests. The similarity threshold is adjustable, overcoming the limitations of traditional string matching.

• Reduced redundant computations: For identical AI requests, the system returns cached data directly, avoiding redundant calls to the LLM.

• Improved performance: By quickly retrieving results from the cache, AI Gateway significantly reduces response times and the load on backend servers. The semantic cache enables "intent-level" responses, greatly improving user satisfaction and experience.

• Expanded scenario coverage: Supports standard scenarios like customer service systems and knowledge base queries, as well as variants of natural language instructions such as "tomorrow's weather" and "weather forecast for the next 24 hours".

• Log monitoring: Allows you to analyze cache hit ratio metrics.

Procedure

1. Log on to the AI Gateway console and choose Instance. In the top menu bar, select a region, then click the target instance ID.

2. In the navigation pane on the left, choose Model API, then click the target API name to go to the API Details page.

3. Click Policies and Plug-ins, enable the Cache switch, and configure the parameters.

   AI Gateway has upgraded its caching capabilities to support Semantic and Exact Match. Select the appropriate cache mode based on the Strategy comparison.

   Semantic cache

   Important

   If a request includes the x-higress-skip-ai-cache: on header, the request bypasses the cache. It is forwarded directly to the backend service, and the response is not cached.

   • Cache Key Strategy: Select the default option, Latest Query Only, or select Integrate Historical Queries based on your requirements.

   • Text Vectorization Configuration:

     • AI Service: Select an existing AI service. If you do not have an AI service, click Create Service to create one.

     • Model Name: Select the name of the model that you want to use.

     • Timeout Period: Set the timeout period. The default value is 5,000 ms.

   • Vector Database Configuration:

     • Service Provider: If you have not activated DashVector, click Go to Console to open the service activation page. You will need to . Record the collection name for later use.

       Important

       When you create a collection, select Cosine as the distance metric, and ensure the vector dimension matches that of the text embedding model. To find the output vector dimensions of text embedding models on the Model Studio platform, see Text and Multimodal Embeddings.

     • Service URL: Enter your DashVector service endpoint.

     • Collection Name: Enter the name of the collection that you created.

     • API Key: The access credential. For more information, see Manage API keys.

     • Vector Similarity Threshold: This value determines how strictly queries are matched to cached content. It ranges from 0 to 1, with a recommended value of 0.8 or 0.9. A higher value indicates greater semantic similarity. For more information, see Vector similarity threshold configuration.

     • Timeout Period: Set the timeout period. The default value is 3,000 ms.

   Precise cache

   Important

   • Go to the Redis console and add the VPC CIDR block of the gateway instance to the whitelist.

   • If a request includes the x-higress-skip-ai-cache: on header, the request bypasses the cache. It is forwarded directly to the backend service, and the response is not cached.

   • Cache Key Strategy: Select the default option, Latest Query Only, or select Integrate Historical Queries based on your requirements.

   • Redis Cache Configuration:

     • Redis service URL: Enter your Redis service URL.

     • Port Number: Enter your port number.

     • Access Method: Select the method for accessing the Redis service. The options are Account+Password, Password-only, and Password-free.

     • Database Account: If you select the Account+Password access method, enter the database account.

     • Database Password: If you select an access method that requires a password, enter the database password.

     • Database No.: The specified database number.

     • Duration (seconds): The default cache duration is 1,800 seconds. During this period, if the API receives an identical AI request, the LLM is not called, and the cached response is returned directly.

4. Confirm the configuration and click Save.

Vector similarity threshold configuration

Cache hit ratio

You can query the cache hit ratio at the gateway level using the following search statement:

cluster_id:{your-gatewayId} and inner-ai-cache-{your-gatewayId} | SELECT 
SUM(CASE WHEN content LIKE '%cache hit for key%' OR content LIKE '%key accepted%' THEN 1 ELSE 0 END) AS hit_count,
SUM(CASE WHEN content LIKE '%cache miss for key%' OR content LIKE '%score not meet the threshold%' THEN 1 ELSE 0 END) AS miss_count,
SUM(CASE WHEN content LIKE '%cache hit for key%' OR content LIKE '%key accepted%' THEN 1 ELSE 0 END) * 100.0 / 
NULLIF(SUM(CASE WHEN content LIKE '%cache hit for key%' OR content LIKE '%key accepted%' OR content LIKE '%cache miss for key%' 
           OR content LIKE '%score not meet the threshold%' THEN 1 ELSE 0 END), 0) AS hit_rate

• Replace {your-gatewayId} with your gateway instance ID. Note that you must retain the gw- prefix for the first replacement, but not for the second. If you access the plug-in log query system from the cache switch button, the query box automatically contains a cluster_id filter. In this case, you only need to paste the rest of the query statement after the filter. On the Monitoring (AI Cache) page, click View Logs to go to the log query page. The log source is apig-plugin-log. Enter the preceding query statement into the query bar and run it.

• The query returns the cache hit ratio.

  For example, the result might show that hit_count (cache hits) is 7, miss_count (cache misses) is 6, and hit_rate (hit ratio) is approximately 53.85%.

Examples

• When precise cache is enabled, only identical queries are matched. For example, if a user sends the query "Who are you?" and it is not served from the cache, the model consumes tokens as usual (input tokens: 127, output tokens: 40). If the same query hits the precise cache (indicated by from-cache), both input and output tokens are 0, and the cached result is returned directly. This significantly reduces token consumption.

• When semantic cache is enabled, semantically similar queries can also be matched. Queries with a semantic similarity below the threshold will not be matched.

  When a semantic cache hit occurs, the cached response is reused for similar questions. Both input and output tokens are 0, consuming no model inference resources.

  For example, if a user asks Where can my package be delivered today?, and this query differs significantly in semantics from the cached questions, a cache miss occurs. The qwen-max model then generates a response and consumes tokens as usual.