Gateways
LLM gateways sit between your application and model providers. They expose one baseURL, route across providers, and meter spend — the layer you optimize when Cline, Aider, OpenCode, or a production agent shares one API key.
These guides apply the practice sequence at the gateway layer. Gateways do not replace context hygiene in your editor — they forward what clients send.
When to optimize here
| Signal | Gateway fix |
|---|---|
| Bill spike with no IDE dashboard | Activity / spend logs on the gateway |
| One API key across dev + CI + prod | Per-key caps and workspace budgets |
| Always billed for frontier model | Haiku-first routing and fallbacks |
cached_tokens always zero | Stabilize prefix; enable response caching |
| Double proxy (gateway + gateway) | Pick one routing layer |
Pair every gateway guide with your IDE guide — trim MCP, rules, and context upstream.
Walk sequence
Every gateway guide follows the same six steps:
- Measure — dashboard, exports, or spend API
- Cap — per-key limits, workspace budgets, rate limits
- Route cheap — fallback chains, price sorting, model allowlists
- Cache — prefix caching, response caching, edge cache
- Trim upstream — IDE context, MCP, rules
- Guardrail in prod — org caps, batch workloads, observability
Gateway comparison
| Gateway | Billing | Self-host | Caching | Caps | Best fit |
|---|---|---|---|---|---|
| OpenRouter | Credits + platform fee | No | Prefix + response | Per-key, workspace | BYOK across 500+ models, coding agents |
| LiteLLM | Your provider bills | Yes (proxy) | Provider-dependent | Virtual keys, budgets | Team proxy over local + cloud backends |
| Vercel AI Gateway | Provider rates, no markup | No | Provider-dependent | Vercel dashboard | Next.js / AI SDK production apps |
| Portkey | Provider rates + platform | No | Gateway cache | Budgets, configs | Enterprise fallback chains + observability |
| Helicone | Credits or BYOK, 0% markup | No | Prompt cache proxy | Rate limits | Request logging + throttle before provider |
| Cloudflare AI Gateway | Provider rates | No | Edge cache | Rate limits | Edge caching, unified analytics |
Not here: LocalAI is an inference gateway over local engines — see Self-hosting.
Guides
| Gateway | Guide | Dominant waste |
|---|---|---|
| OpenRouter | Tokenminning with OpenRouter | Frontier defaults, missing fallbacks, unstable prefixes |
| LiteLLM | Tokenminning with LiteLLM | Unscoped virtual keys, no spend tracking DB |
| Vercel AI Gateway | Tokenminning with Vercel AI Gateway | Frontier defaults in AI SDK, no spend monitoring |
| Portkey | Tokenminning with Portkey | Missing budget configs, unbounded fallback chains |
| Helicone | Tokenminning with Helicone | No rate limits, unlogged agent loops |
| Cloudflare AI Gateway | Tokenminning with Cloudflare AI Gateway | Cache misses, no rate limits at edge |
Anti-patterns (all gateways)
| Anti-pattern | Why it fails |
|---|---|
| Gateway + gateway stack | Latency and fees compound |
| One key for CI and dev | Overnight agent loops drain pool |
| Frontier default on every call | 60–90% savings left on table from routing |
| No spend export | Cannot attribute by feature or team |
Related
Last updated on