Text Generation Inference (TGI): deployment and alternatives
TGI is Hugging Face’s model-serving runtime with continuous batching and an OpenAI-compatible chat API. For an existing deployment, start with the server guide and check the supported-model list for your checkpoint’s architecture.
The upstream repository describes TGI as being in maintenance mode and recommends vLLM or SGLang for future development. A familiar --model-id flag does not establish support for a new architecture or weight format.
When to choose TGI
| Signal | What it means |
|---|---|
| Existing Kubernetes + HF Hub models | TGI targets containerized Hub checkpoints at scale |
| Enterprise HF support path | You already operate TGI in production |
| Greenfield CUDA serving | Prefer vLLM or SGLang for new GPU stacks |
For Docker quick start and runtime terminology, see text generation inference.
Setup and operation
| Guide | What you’ll learn |
|---|---|
| Install the server and send a request | Container run, first completion |
| Size GPU memory | VRAM for context and batching |
| Connect an API client | OpenAI-compatible chat API |
| Deploy on Kubernetes | Production layout |
| Monitor serving | Metrics and health checks |
| Calculate costs | Tokens vs GPU time |
| Apply request guardrails | Limits and safety |
| Find support options | Upgrades and vendor paths |
Model-specific deployment guides
No model pages document a TGI recipe for these checkpoints yet. Use the model directory for vLLM, SGLang, and iwant paths, and confirm your checkpoint on the TGI supported-model list.
Each guide links the actual upstream configuration, explains hardware and checkpoint variants, and includes a first API request. Browse all documented models for other deployment options.
TGI alternatives
| Tool | Choose when |
|---|---|
| iwant | You want a documented model name to become a GCP endpoint without assembling VMs, engine flags, and teardown yourself. |
| vLLM | You manage the GPU host and need continuous batching, prefix caching, and direct control over vllm serve settings. |
| SGLang | You need RadixAttention prefix reuse, structured generation, or model-specific parsers on hardware you configure. |
| Ollama | You want local ollama pull ergonomics, IDE chat, and low-ops serving on a single machine. |
| llama.cpp | You serve GGUF weights on CPU, Apple Silicon, or a modest GPU with minimal dependencies. |
| LocalAI | You need one OpenAI-compatible URL over multiple backends or gallery configurations. |
All launch methods: self-hosting hub.
Related reading
- Self-hosting hub — economics, model directory, and launch comparison
- Text generation inference — runtime terminology and workload fit
- Local inference practice — when to self-host and how to meter GPU time