LiteLLM model routing

LiteLLM routing sends requests to the cheapest model that succeeds — via router_settings, model groups, and fallback lists in config.yaml.

Hub: Tokenminning with LiteLLM. Pattern: Model routing.

Fallback pattern in config.yaml

model_list:
  - model_name: agent-routine
    litellm_params:
      model: gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: agent-escalation
    litellm_params:
      model: gpt-4o
      api_key: os.environ/OPENAI_API_KEY
 
router_settings:
  routing_strategy: simple-shuffle
  model_group_alias:
    agent: [agent-routine, agent-escalation]

Clients request model: agent — LiteLLM tries agent-routine first, falls through on failure.

Tier selection

Route to local backends

Point model entries at local Ollama or vLLM for zero API cost on routine work:

- model_name: local-routine
  litellm_params:
    model: ollama/llama3.2
    api_base: http://localhost:11434

Hybrid pattern: local for routine, cloud for escalation — see local inference.

Wire to tools

Use virtual keys per tool — budgets.

Spending guardrails

• Set models allowlist on CI virtual keys
• Default agent-routine to mini/haiku tier
• Log x-litellm-response-cost header in your middleware
• Avoid routing everything through one frontier alias

Related

• LiteLLM hub
• Budgets & spend
• Caching
• Routing docs