Integrate the semantic cache with a self-hosted agent

Mode comparison

The following table highlights the key differences between OpenAI-compatible mode and LangCache-compatible mode.

Prerequisites

Step 1: Obtain connection information

1. Log on to the Tair console. In the navigation pane on the left, click Tair Semantic Cache Gateway Instance.

2. In the instance list, click the name of your target instance to open its details page.

3. In the Access Information section, obtain the Base URL and API Key.

4. Go to the Plugin Management tab and obtain the Cache ID from the semantic cache plugin section.

   Information	Purpose	Format
   Endpoint / Base URL	The target address for API requests.	https://tg-bp1******.redis.rds.aliyuncs.com
   API Key	Used for authentication.	sk-****
   Cache ID	Identifies the cache instance and is used in the API path.	r-bp1xxxxxx

Step 2: Understand the integration logic

Implement the following workflow in your agent's code:

User asks a question
   │
   ▼
Search cache (POST /entries/search)
   │
   ├── Cache hit → Return the cached response (millisecond-level)
   │
   └── Cache miss → Call your own LLM to get a response
                    │
                    ▼
              Insert into cache (POST /entries)
                    │
                    ▼
              Return the LLM response to the user

This process uses two core APIs. For details about all available APIs, see Architecture and API Reference.

Step 3: Implement the integration code

The following example shows the complete integration process using the Python SDK.

pip install langcache

from langcache import LangCache
from openai import OpenAI  # Example of using the OpenAI SDK for LLM calls

# ============ Configuration ============
LANGCACHE_BASE_URL = "https://tg-xxxxxx.redis.rds.aliyuncs.com"
LANGCACHE_API_KEY  = "sk-xxxxxx"
CACHE_ID           = "r-bp1xxxxxx"

# Your LLM configuration (used on a cache miss)
LLM_API_KEY  = "<Your-LLM-API-Key>"
LLM_BASE_URL = "<Your-LLM-Base-URL>"
LLM_MODEL    = "<Your-LLM-Model>"

# ============ Initialize clients ============
cache_client = LangCache(
    server_url=LANGCACHE_BASE_URL,
    api_key=LANGCACHE_API_KEY,
    cache_id=CACHE_ID,
)

llm_client = OpenAI(
    api_key=LLM_API_KEY,
    base_url=LLM_BASE_URL,
)

# ============ Core logic ============
def ask(question: str) -> str:
    """A Q&A function with semantic cache."""

    # Step 1: Search the cache
    search_result = cache_client.search(prompt=question)

    if search_result.data:
        # Cache hit, return the response directly
        print(f"[Cache Hit] Similarity: {search_result.data[0].similarity}")
        return search_result.data[0].response

    # Step 2: Cache miss, call the LLM
    print("[Cache Miss] Calling LLM...")
    completion = llm_client.chat.completions.create(
        model=LLM_MODEL,
        messages=[{"role": "user", "content": question}],
    )
    answer = completion.choices[0].message.content

    # Step 3: Insert the question-answer pair into the cache
    cache_client.set(prompt=question, response=answer)

    return answer


# ============ Example usage ============
if __name__ == "__main__":
    # First question (calls the LLM)
    print(ask("What is semantic cache?"))

    # Semantically similar question (hits the cache, returns in milliseconds)
    print(ask("What does semantic cache mean?"))

Step 4: Verify the cache

After integrating the code, verify that the cache works as expected:

1. Send a question (such as What is Redis Cluster) and observe whether the LLM returns a normal response.

2. Send a question that is semantically similar but phrased differently (such as Redis Cluster - what is it), and observe the following:

   • The response content is identical to the first response.

   • The response time is significantly faster (reduced from tens of seconds to millisecond-level latency).

If the second request returns the same content much faster, the semantic cache is working correctly.

Usage notes

Related operations

• Use Attributes to implement multi-tenant isolation: Attach the attributes field, such as {"tenant_id": "xxx"}, when you insert and search for data. Cache data with different attributes values are isolated from each other. Note: The attribute field names must be declared in advance when you create the Cache.

• Set cache expiration time (TTL): Set an expiration time for a single entry in milliseconds by using the ttlMillis parameter during insertion, or configure a global TTL in the console.

• Delete a single cache entry: Use DELETE /v1/caches/{cacheId}/entries/{entryId} to delete an outdated or incorrect cache entry.

• Flush all caches: When old answers are no longer valid after a model upgrade, use POST /v1/caches/{cacheId}/flush to rebuild the cache (this action is irreversible).

• Adjust the similarity threshold: A lower threshold increases the hit rate but may result in false matches. A higher threshold improves precision but reduces the hit rate.

• API reference: See Architecture and API Reference.