LocalAI: self-host an API with multiple model backends
LocalAI provides an OpenAI-compatible API for model backends and gallery configurations. Use it when you want one client configuration for several model types. The chosen backend and model configuration determine memory requirements, supported features, and GPU behavior. See the official documentation for current platform and model support.
When to choose LocalAI
| Signal | What it means |
|---|---|
One baseURL for many backends | Route different models through the same OpenAI-compatible gateway |
| Gallery-driven setup | Prebuilt configurations instead of hand-written engine flags |
| Hybrid local + remote | Some backends proxy to cloud APIs while others run locally |
The gateway adds operational surface — you still size and lifecycle each backend underneath.
Setup and operation
| Guide | What you’ll learn |
|---|---|
| Install LocalAI | Containers, models path, first start |
| Choose a backend | llama.cpp, vLLM, and gallery options |
| Connect an OpenAI client | Endpoints, keys, and model names |
Model-specific deployment guides
Each guide links the actual upstream configuration, explains hardware and checkpoint variants, and includes a first API request. Browse all documented models for other deployment options.
LocalAI alternatives
| Tool | Choose when |
|---|---|
| iwant | You want a documented model name to become a GCP endpoint without assembling VMs, engine flags, and teardown yourself. |
| vLLM | You manage the GPU host and need continuous batching, prefix caching, and direct control over vllm serve settings. |
| SGLang | You need RadixAttention prefix reuse, structured generation, or model-specific parsers on hardware you configure. |
| TGI | You already run Hugging Face Text Generation Inference on Kubernetes and the checkpoint is on the supported-model list. |
| Ollama | You want local ollama pull ergonomics, IDE chat, and low-ops serving on a single machine. |
| llama.cpp | You serve GGUF weights on CPU, Apple Silicon, or a modest GPU with minimal dependencies. |
All launch methods: self-hosting hub.
Related reading
- Self-hosting hub — economics, model directory, and launch comparison
- Text generation inference — runtime terminology and workload fit
- Local inference practice — when to self-host and how to meter GPU time
Last updated on