Tokenminning with LiteLLM
LiteLLM is an open-source LLM proxy — one OpenAI-compatible baseURL that routes to 100+ providers and local backends (vLLM, Ollama). Teams self-host it for spend tracking, virtual keys, and budget caps.
Hub for the LiteLLM cluster. Pair with your IDE guide and Zed local models if you route through LiteLLM.
Quick checklist
- Enable the spend tracking database — LiteLLM needs Postgres for metering.
- Issue virtual keys per developer and environment with
max_budget. - Route routine work to cheap models — routing.
- Stabilize prefixes for provider-side cache hits — caching.
- Point clients at one proxy URL — proxy setup.
Typical impact: 50–90% routing routine steps to mini-tier; virtual key caps prevent overnight agent loops.
How to reduce LiteLLM spending
1. Measure first
Open the LiteLLM UI at /ui or query /global/spend/report. Check x-litellm-response-cost in response headers per request.
2. Cap spend
Generate virtual keys with max_budget and budget_duration. Set team-level budgets in the admin UI.
3. Route cheap
Configure router_settings with fallback models. Default haiku/mini for agent loops; escalate on failure.
4. Cache prefixes
LiteLLM passes through provider caching. Stabilize system prompts upstream; read cached_tokens in provider responses.
→ Caching
5. Trim upstream bloat
LiteLLM forwards what clients send. Trim MCP and rules in OpenCode, Cline, or Aider.
6. Guardrail in prod
Tag requests with metadata.tags for cost attribution. Block CI keys from frontier models via models allowlist on the key.
How LiteLLM bills
LiteLLM itself is free (open source). You pay upstream provider rates — Anthropic, OpenAI, Azure, or your local GPU time. LiteLLM calculates cost from its model cost map .
Guides
| Guide | What you’ll learn |
|---|---|
| Budgets & spend | Virtual keys, max_budget, spend reports |
| Model routing | Fallbacks, router settings, model lists |
| Caching | Provider cache passthrough, prefix stability |
| Proxy setup | config.yaml, client wiring, Zed integration |
Anti-patterns
| Anti-pattern | Why it fails |
|---|---|
| Master key shared across team | No attribution; one runaway loop drains all |
| LiteLLM + OpenRouter double proxy | Latency and fees stack |
| No database for spend tracking | Cannot cap or report |
Frontier default in config.yaml | Every agent step at max tier |