Zed local models

Zed supports local inference through Ollama and custom OpenAI-compatible endpoints. GSC queries like zed llama.cpp, zed vllm, zed litellm, and zed rag usually mean wiring a local stack instead of cloud APIs.

Hub: Tokenminning in Zed. Stack guides: Ollama, llama.cpp, vLLM.

Ollama (fastest path)

ollama pull llama3.2

In Zed assistant settings, point the provider to your Ollama baseURL (default http://localhost:11434). Zed sends OpenAI-compatible requests — Ollama runs llama.cpp under the hood.

Use for: exploration, air-gapped work, drafts before cloud commit.

llama.cpp server

Run llama-server and set Zed’s provider base URL to http://localhost:8080/v1. Same pattern as Ollama but you control GGUF paths and --fit-target memory sizing.

vLLM

For a shared local vLLM instance on a dev GPU:

• baseURL: http://localhost:8000/v1
• Pick a model name the server exposes
• Cap concurrent Zed users if the GPU is shared

zed vllm queries are usually about this OpenAI-compatible URL swap — not a Zed-specific vLLM plugin.

LiteLLM gateway — Zed → LiteLLM → provider

zed litellm setups route Zed through a LiteLLM OpenAI-compatible proxy so one baseURL fans out to many providers, local backends, or spend-capped routes.

Traffic path:

Zed assistant → LiteLLM proxy (:4000/v1) → upstream provider or local engine

1. Start LiteLLM with a config that lists your models (LiteLLM proxy):

litellm --config /path/to/litellm_config.yaml --port 4000

2. In Zed Settings → Assistant → OpenAI-compatible provider, set:

3. Send a test prompt in Zed and confirm the model in LiteLLM logs.

Example litellm_config.yaml fragment:

model_list:
  - model_name: dev-cheap
    litellm_params:
      model: gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: local-llama
    litellm_params:
      model: ollama/llama3.2
      api_base: http://localhost:11434

Useful for team routing and spend caps — adds latency; avoid double gateways (LiteLLM + OpenRouter) unless required.

RAG and retrieval

zed rag is not a separate Zed product feature — it means assistant retrieval over your codebase or attached context. Prefer:

• Narrow @file / @symbol references — Context guide
• Trim indexed surface before adding external RAG pipelines
• For app-level RAG, see Output and RAG

Hybrid routing

Log which backend served each task for cost attribution.

Related

• Zed hub
• Token usage
• Local inference