Text Generation Inference (TGI)
Text Generation Inference — text-generation-inference on GitHub — is Hugging Face’s production inference engine for transformer models. TGI targets teams whose models live on the Hugging Face Hub and whose deployment target is Kubernetes at scale.
Choose TGI when you need Hub-native loading, OpenAI-compatible APIs, and a path to managed Inference Endpoints . For laptop dev servers, use Ollama or llama.cpp.
Expected impact
| Scenario | Typical outcome |
|---|---|
| Steady production traffic on HF models | 60–90% vs cloud API at high GPU utilization |
| Sporadic internal experiments | Negative — cluster overhead dominates |
| Models requiring HF-specific features (flash attention, custom code) | TGI reduces glue code vs rolling your own vLLM config |
Benchmark against vLLM on your hardware — both are strong; the winner is workload-specific.
Architecture
Client → Ingress / API Gateway
↓
TGI router (K8s Service)
↓
TGI replicas (continuous batching)
↓
GPU node poolTGI handles continuous batching, tensor parallelism, quantization (bitsandbytes, GPTQ, AWQ depending on version), and an OpenAI-compatible chat completions endpoint at http://localhost:8080/v1/chat/completions.
Guides
| Guide | What you’ll learn |
|---|---|
| Server | Run the TGI server with Docker — model args, memory limits, first request |
| API | OpenAI-compatible endpoints, request shapes, client SDK wiring |
| Kubernetes | Production deploys — PVCs, scaling, secrets, blue/green upgrades |
| Observability | Prometheus metrics, span tags, queue-depth alerts |
| Guardrails | Progressive enforcement — soft alerts through hard token caps |
| Support | Enterprise support paths, Inference Endpoints, HF token management |
TGI vs vLLM
| Factor | TGI | vLLM |
|---|---|---|
| HF Hub integration | Native | Supported via model paths |
| Ecosystem momentum | HF enterprise path | Broad OSS community |
| Feature velocity | Tied to HF releases | Fast iteration on CUDA kernels |
| Managed option | Inference Endpoints | Third-party hosts |
Run both on your eval set and pick on measured throughput, latency, and quality — not brand preference.
Anti-patterns
| Anti-pattern | Why it fails |
|---|---|
| TGI for a single developer laptop | Docker + GPU passthrough friction for no gain |
max-input-length at model maximum for all routes | KV cache exhaustion; terrible batching |
| No HF token rotation | Gated model pulls fail silently in CI |
| One giant deployment for unrelated model tiers | Noisy neighbor latency; cannot scale tiers independently |
Related
- Self-hosting — when on-prem beats cloud
- vLLM — alternative production engine
- Output and RAG — trim retrieval before TGI sees it
- TGI documentation