Text generation inference

Text generation inference is serving open-weight language models on hardware you control — turning prompts into completions through a local or on-prem runtime instead of a per-token cloud API.

For launch steps and model-specific commands, start at the self-hosting hub (model directory + launch-tool comparison). This page explains runtime terminology and workload fit; each engine has a matching hub under /self-hosting/{iwant,vllm,sglang,tgi,ollama,llama-cpp,localai}.

Need Hugging Face TGI operations? Follow the TGI hub after the Docker quick start below.

TGI quick start (Hugging Face)

If you landed here for text generation inference Docker setup:

docker run --gpus all --shm-size 1g -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct \
  --max-input-length 4096 \
  --max-total-tokens 8192

Next steps: TGI hub → hardware sizing → cost per token → OpenAI-compatible API.

Pick a runtime (inference engine or gateway) by workload profile and utilization, not by which IDE shipped a one-click installer. Start with Local inference for the practice sequence; use the self-hosting hub to choose a launch method, then open the deep dive for setup and guardrails.

Runtimes at a glance

All listed runtimes expose OpenAI-compatible HTTP endpoints — your SDK or router can swap baseURL without rewriting client code.

Deployment automation

iwant

iwant turns a documented model name into a cloud endpoint: it provisions the GPU host, configures serving, checks readiness, and prints a URL and key. GCP is currently supported. Start with the model directory to compare its automated launch with model-specific vLLM, SGLang, and local runtime instructions.

vLLM

vLLM  is a production inference engine built for throughput. PagedAttention and continuous batching let one GPU serve many concurrent requests efficiently — the stack teams reach for when Ollama’s single-user ergonomics stop scaling.

vLLM pays off when utilization is high and model choice is stable. Sporadic dev experiments belong on Ollama.

→ vLLM deep dive

SGLang

SGLang  is the other greenfield CUDA engine Hugging Face recommends beside vLLM: RadixAttention for prefix reuse, strong structured generation, and OpenAI-compatible serving. Pair with benchmarks against vLLM on your concurrency mix.

→ SGLang deep dive

TGI (Text Generation Inference)

TGI  is Hugging Face’s production serving stack for transformer models. It targets teams already on the HF Hub who want OpenAI-compatible APIs, Kubernetes-native scaling, and enterprise support paths.

Choose TGI when models live on Hugging Face and deployment target is Kubernetes at scale — not when you want a laptop dev server.

→ TGI deep dive — hardware sizing, cost per token

Ollama

Ollama  is the fastest path from a downloaded model to a working local assistant. One binary, ollama pull, and an OpenAI-compatible server. Most teams meet Ollama through IDE integrations (Zed, Aider, Cline) before they consider production serving.

Ollama optimizes for developer experience, not maximum tokens per second. Use it for exploratory coding, dev GPUs, and hybrid routing — not multi-tenant production at datacenter scale.

→ Ollama deep dive

llama.cpp

llama.cpp  is a C/C++ inference runtime for GGUF-quantized models. It runs on CPUs, Apple Silicon, and modest GPUs with minimal dependencies — the engine inside many “local AI” tools, including Ollama and LocalAI.

Use llama.cpp when you need portable, low-footprint inference — not datacenter-scale batching.

→ llama.cpp deep dive

LocalAI

LocalAI  is a self-hosted inference gateway — not a single engine. It exposes an OpenAI-compatible API and can front llama.cpp, vLLM, diffusers, whisper, and optional cloud backends behind one URL.

LocalAI’s value is integration surface area, not raw tokens per second. At 100+ QPS, put vLLM or TGI directly behind your router instead of adding a gateway hop.

→ LocalAI deep dive

Workload fit

Instrumentation and guardrails

Self-hosted inference still needs budgets — measured in GPU hours and queue depth, not procurement theater.

See Article IV: fiscal ceilings for runtime cap patterns that apply to both cloud and on-prem.

Related

• Self-hosting — cost model and when on-prem beats cloud
• Local inference — practice guide (step 8 in the optimization stack)
• Model routing — hybrid local/cloud escalation