Skip to Content
GatewaysVercel AI GatewayCaching

Vercel AI Gateway caching

Vercel AI Gateway is a routing layer — prefix caching comes from upstream providers when your AI SDK calls send stable system prompts.

Hub: Tokenminning with Vercel AI Gateway. Concept: Prompt caching.

Stabilize system prompts

import { generateText } from "ai";
 
const result = await generateText({
  model: "anthropic/claude-3-5-haiku-20241022",
  system: STATIC_SYSTEM_PROMPT, // pin across turns
  prompt: userMessage,
});

Do not regenerate the system prompt with timestamps or dynamic state every turn.

Read cache metrics

Inspect usage in AI SDK responses. For Anthropic models routed through the gateway, look for cached_tokens in provider metadata.

Multi-turn agents

For agent loops with streamText or tool calling:

  1. Pin tool definitions in the system message
  2. Append user/assistant turns — do not reshuffle
  3. Check cache hit rate after turn 3

Trim tool schemas — fewer tools means smaller prefixes and better cache stability.

vs semantic caching

Semantic caching is application middleware — not built into AI Gateway. Add it before the gateway call if you need similarity-based dedup.

Spending guardrails

  • Compare token usage before/after stabilizing system prompts
  • Route cache-friendly workloads to Anthropic/OpenAI (providers with prefix caching)
  • Monitor observability for input token trends per feature
Last updated on
Model routingPortkey