LocalAI backends – Tokenminning

Each LocalAI model YAML picks a backend engine. The gateway exposes one OpenAI API; backends determine throughput, memory, and model format.

Hub: LocalAI.

Backend selection

Backend	Engine	Best for
`llama-cpp`	llama.cpp	CPU/Metal, GGUF weights
`vllm`	vLLM	GPU throughput, continuous batching
`ollama`	Ollama	Delegate to running Ollama daemon
`transformers`	Hugging Face	Compatibility; slower than vLLM/TGI

llama-cpp example


name: llama3
backend: llama-cpp
parameters:
  model: models/llama-3.2-8b-instruct.Q4_K_M.gguf
  context_size: 8192
  gpu_layers: 99

vLLM backend

Point LocalAI at an existing vLLM OpenAI server or configure in-process vLLM per LocalAI docs. Use when you need gateway features (embeddings + chat) but want vLLM throughput for chat.

Direct vLLM at high QPS avoids LocalAI hop latency — see When to skip LocalAI.

Ollama backend

Reuse ollama pull models without duplicating weights:


name: llama3-ollama
backend: ollama
parameters:
  model: llama3.2

Requires Ollama running and reachable from the LocalAI container.

Hybrid routing

Route features to backends via separate model name values:


classify → localai model "llama3-small" (llama-cpp Q4)
legal-review → localai model "cloud-proxy" (external API backend)

Log inference.backend on every span for cost attribution.

Anti-patterns

Pattern	Why it fails
LocalAI → LocalAI chains	Latency stacks
One 70B for all names	Cannot route cheap tasks to small models
Ignoring per-backend memory	llama.cpp OOM kills worker