Semantic caching

Semantic caching (also called a semantic cache) stores past LLM responses keyed by query meaning. On a new request, you embed the query, search a vector index for a near-duplicate, and return the stored answer without calling the model when similarity exceeds your threshold.

This page answers what is semantic caching, how does semantic caching work, and prompt caching vs semantic caching.

This is a Tier 3 optimization. Ship metering, prompt hygiene, and provider prompt caching first.

How semantic caching works

Typical request path:

1. User query arrives
2. Embed query text → vector (same model family as index)
3. Vector DB nearest-neighbor search (cosine / dot product)
4. If similarity ≥ threshold → return cached response (no LLM)
5. Else → call LLM → store (query embedding, response) on success

Semantic cache hits skip generation entirely — unlike prompt caching, which only discounts re-processing a stable prefix.

Minimal middleware pattern

async function handleQuery(userText: string) {
  const embedding = await embed(userText);
  const hit = await vectorIndex.search(embedding, { topK: 1, minScore: 0.92 });
 
  if (hit) {
    metrics.increment("semantic_cache.hit");
    return hit.response;
  }
 
  const response = await llm.complete({ messages: [...] });
  await vectorIndex.upsert({
    id: hash(userText),
    vector: embedding,
    response,
    ttlSeconds: 3600,
  });
  metrics.increment("semantic_cache.miss");
  return response;
}

Tag spans with cache.layer=semantic and cache.hit=true|false so you can compare against prefix cache metrics on the same route.

Libraries such as GPTCache  wrap embed + store + lookup; you can also use Redis with vector search or a dedicated vector DB.

Prompt caching vs semantic caching

Use both in a stack: exact string cache → semantic cache → provider prefix cache → full LLM (see architecture below).

OpenRouter and other gateways expose prefix metrics as cached_tokens — semantic hits show up in your middleware metrics, not provider usage fields. See OpenRouter caching.

Expected impact

Semantic caching eliminates the full inference call on cache hits — saving 100% of generation tokens for matched queries. ROI depends entirely on query repetition:

When to use semantic caching

Good fit:

• High-traffic endpoints with repetitive queries
• Support bots answering common questions
• Classification tasks with finite input patterns
• Tool-result lookups that change infrequently

Poor fit:

• Unique creative generation per request
• Real-time data queries (prices, inventory, news)
• Tasks where a wrong cached answer is worse than no answer
• Low-traffic endpoints (cache overhead exceeds savings)

Architecture: multi-tier caching

A layered approach covers the most ground:

Request
  ↓
┌─────────────────────┐
│ Tier 1: Exact match │  ← identical query string (sub-ms)
└─────────┬───────────┘
          ↓ miss
┌─────────────────────┐
│ Tier 2: Semantic    │  ← vector similarity (low-ms)
└─────────┬───────────┘
          ↓ miss
┌─────────────────────┐
│ Tier 3: Provider    │  ← prefix cache (see prompt caching guide)
│         prefix cache│
└─────────┬───────────┘
          ↓ miss
┌─────────────────────┐
│ Tier 4: Full LLM    │  ← inference call
│         inference   │
└─────────────────────┘

Each tier catches queries the previous tier missed. Cache the LLM response at Tier 2 for future semantic matches.

Similarity thresholds

Set thresholds based on the cost of a wrong answer:

Below 0.85, you risk returning cached responses to queries that are semantically adjacent but factually different. A wrong cached answer is worse than no cache.

TTL and staleness

Cached responses do not know when their source data has changed. Set aggressive TTL for dynamic data:

For data that changes frequently, semantic caching may cause more harm than benefit. Provider prompt caching is a better fit for stable prefixes with dynamic suffixes.

Measuring success

Emit semantic_cache.hit_rate, semantic_cache.stale_returns, and compare cost per outcome — not raw hit rate alone.

Related

• Prompt caching — prefix / KV reuse inside the provider
• Context hygiene — trim dynamic suffix before any cache layer
• Output and RAG — when retrieval replaces memorization
• Where to start — technique order in the stack