Text generation inference

Text generation inference is serving open-weight language models on hardware you control — turning prompts into completions through a local or on-prem runtime instead of a per-token cloud API.

Pick a runtime (inference engine or gateway) by workload profile and utilization, not by which IDE shipped a one-click installer. Start with Local inference for the practice sequence; use this page to choose a stack, then open the deep dive for setup and guardrails.

Runtimes at a glance

All five expose OpenAI-compatible HTTP endpoints — your SDK or router can swap baseURL without rewriting client code.

Runtime profiles

Ollama

Ollama  is the fastest path from a downloaded model to a working local assistant. One binary, ollama pull, and an OpenAI-compatible server. Most teams meet Ollama through IDE integrations (Zed, Aider, Cline) before they consider production serving.

Ollama optimizes for developer experience, not maximum tokens per second. Use it for exploratory coding, dev GPUs, and hybrid routing — not multi-tenant production at datacenter scale.

→ Ollama deep dive

vLLM

vLLM  is a production inference engine built for throughput. PagedAttention and continuous batching let one GPU serve many concurrent requests efficiently — the stack teams reach for when Ollama’s single-user ergonomics stop scaling.

vLLM pays off when utilization is high and model choice is stable. Sporadic dev experiments belong on Ollama.

→ vLLM deep dive

llama.cpp

llama.cpp  is a C/C++ inference runtime for GGUF-quantized models. It runs on CPUs, Apple Silicon, and modest GPUs with minimal dependencies — the engine inside many “local AI” tools, including Ollama and LocalAI.

Use llama.cpp when you need portable, low-footprint inference — not datacenter-scale batching.

→ llama.cpp deep dive

TGI (Text Generation Inference)

TGI  is Hugging Face’s production serving stack for transformer models. It targets teams already on the HF Hub who want OpenAI-compatible APIs, Kubernetes-native scaling, and enterprise support paths.

Choose TGI when models live on Hugging Face and deployment target is Kubernetes at scale — not when you want a laptop dev server.

→ TGI deep dive

LocalAI

LocalAI  is a self-hosted inference gateway — not a single engine. It exposes an OpenAI-compatible API and can front llama.cpp, vLLM, diffusers, whisper, and optional cloud backends behind one URL.

LocalAI’s value is integration surface area, not raw tokens per second. At 100+ QPS, put vLLM or TGI directly behind your router instead of adding a gateway hop.

→ LocalAI deep dive

Workload fit

Decision flow

Instrumentation and guardrails

Self-hosted inference still needs budgets — measured in GPU hours and queue depth, not procurement theater.

See Article IV: fiscal ceilings for runtime cap patterns that apply to both cloud and on-prem.

Related

• Self-hosting — cost model and when on-prem beats cloud
• Local inference — practice guide (step 8 in the optimization stack)
• Model routing — hybrid local/cloud escalation