Skip to Content
GatewaysLiteLLMCaching

LiteLLM caching

LiteLLM is a proxy — it does not add its own cache layer. Prefix caching comes from upstream providers (Anthropic, OpenAI) when your client sends stable prefixes.

Hub: Tokenminning with LiteLLM. Concept: Prompt caching.

Read cache metrics

LiteLLM passes through provider usage fields. After a multi-turn session, check responses for:

{
  "usage": {
    "prompt_tokens_details": {
      "cached_tokens": 8000
    }
  }
}

LiteLLM’s spend logs record total tokens — compare cached_tokens in raw provider responses for cache effectiveness.

Stabilize the prefix

  1. Static system prompt and tool schemas first
  2. Dynamic user content last
  3. Do not reshuffle messages between turns

Trim MCP and rules in your IDE before chasing cache flags — OpenCode MCP.

LiteLLM-specific notes

SettingEffect
cache_control in requestPassed to Anthropic-compatible providers
Model alias changes mid-sessionCache miss — stick to one alias per conversation
Router fallback on cache missMay switch provider; cache does not transfer

vs semantic caching

Semantic caching is an application layer — not built into LiteLLM. Add semantic caching middleware before the proxy if you need similarity-based dedup.

Spending guardrails

  • Inspect cached_tokens after turn 3 in stable agent chats
  • Compare Activity in LiteLLM spend logs before/after stabilizing rules
  • Route cache-friendly workloads to providers with prefix caching (Anthropic, OpenAI)
Last updated on
Model routingProxy setup