vLLM prefix caching
vllm enable-prefix-caching reuses KV cache blocks when requests share an identical prompt prefix — vLLM’s version of provider prompt caching.
Hub: vLLM.
Enable
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 --port 8000 \
--max-model-len 8192 \
--enable-prefix-cachingRequirements
| Requirement | Why |
|---|---|
| Stable prefix first | System + tools before dynamic user content |
| Same model | Prefix cache tied to weights |
| Fixed RAG system block | High hits on repeated templates |
vs semantic caching
| Layer | Matches | Saves |
|---|---|---|
| vLLM prefix cache | Identical prefix bytes | KV recompute |
| Semantic caching | Similar meaning | Full inference skip |
Troubleshooting
Hits stay zero — client reshuffles system prompt each turn; stabilize rules/MCP schemas like OpenRouter caching.
Check Prometheus metrics and TTFT on turn 2+ in agent loops.
Related
Last updated on