Ollama: run local models and serve an API
Ollama downloads packaged models, opens local chat sessions, and serves an OpenAI-compatible API on port 11434. Start with a library entry that fits your machine. A cloud-only catalog entry does not provide local model weights. See the official documentation for current platform and model support.
When to choose Ollama
| Signal | What it means |
|---|---|
| Solo dev or IDE chat | Low setup overhead; models stay on your machine |
| Spiky experiments | No cloud VM to tear down when you stop for the day |
OpenAI-compatible baseURL | Port 11434 works with many SDKs and IDE integrations |
Move to vLLM or SGLang when concurrent API traffic outgrows a single-user local server.
Setup and operation
| Guide | What you’ll learn |
|---|---|
| Install and run a model | ollama pull, chat, memory fit |
| Connect an OpenAI client | /v1 endpoints and model IDs |
| Connect your IDE | Zed, Aider, and other local assistants |
Local serving still has a memory and idle-time cost — see local inference for metering patterns.
Model-specific deployment guides
Each guide links the actual upstream configuration, explains hardware and checkpoint variants, and includes a first API request. Browse all documented models for other deployment options.
Ollama alternatives
| Tool | Choose when |
|---|---|
| iwant | You want a documented model name to become a GCP endpoint without assembling VMs, engine flags, and teardown yourself. |
| vLLM | You manage the GPU host and need continuous batching, prefix caching, and direct control over vllm serve settings. |
| SGLang | You need RadixAttention prefix reuse, structured generation, or model-specific parsers on hardware you configure. |
| TGI | You already run Hugging Face Text Generation Inference on Kubernetes and the checkpoint is on the supported-model list. |
| llama.cpp | You serve GGUF weights on CPU, Apple Silicon, or a modest GPU with minimal dependencies. |
| LocalAI | You need one OpenAI-compatible URL over multiple backends or gallery configurations. |
Deeper comparison: Alternatives to Ollama. All launch methods: self-hosting hub.
Related reading
- Self-hosting hub — economics, model directory, and launch comparison
- Text generation inference — runtime terminology and workload fit
- Local inference practice — when to self-host and how to meter GPU time
- Alternatives to Ollama
- Zed local models
Last updated on