TGI hardware and memory

Text Generation Inference memory use splits into three buckets: model weights, KV cache for active requests, and framework overhead. Most OOM failures come from sizing weights correctly but leaving no room for concurrent requests — not from picking the wrong GPU brand.

Use this page when Search Console (or your own benchmarks) show impressions for “TGI memory requirements,” “text generation inference GPU,” or “fit Llama on H100.” For install flags see TGI server; for production node pools see Kubernetes.

VRAM formula

Component	Rule of thumb
Weights	~2 bytes/param (BF16/FP16), ~1 byte (FP8), ~0.5 bytes (INT4/GPTQ/AWQ)
KV cache	Grows with `max-input-length`, concurrent requests, and batch size
Overhead	Budget 10–20% above weights + KV for CUDA context, activations, and TGI buffers

Example: Llama 3.1 8B in BF16 needs ~16 GB weights. With 8K context and a few concurrent requests, plan for 24–32 GB total — an A10G 24 GB is tight; A100 40 GB is comfortable.

Model × GPU fit table

Starting points for single-replica TGI on one GPU. Adjust --max-input-length and --max-total-tokens down if you need more concurrency.

Model (Hub id)	Precision	Weight VRAM (approx.)	Fits on	Notes
`meta-llama/Llama-3.2-3B-Instruct`	BF16	~6 GB	RTX 4090 24 GB, L4	Dev and smoke tests
`meta-llama/Llama-3.1-8B-Instruct`	BF16	~16 GB	A10G 24 GB (tight)	`--max-input-length 4096` on 24 GB cards
`meta-llama/Llama-3.1-8B-Instruct`	GPTQ/AWQ	~5–6 GB	RTX 4090 24 GB	Use `--quantize gptq` or `awq` when supported
`mistralai/Mistral-7B-Instruct-v0.3`	BF16	~14 GB	A10G / L40S	Common production 7B tier
`meta-llama/Llama-3.3-70B-Instruct`	FP8	~70–75 GB	H100 80 GB (tight)	Lower context or `--num-shard 2` for comfort
`meta-llama/Llama-3.3-70B-Instruct`	BF16	~140 GB	2× H100 80 GB	`--num-shard 2` tensor parallelism
`Qwen/Qwen2.5-32B-Instruct`	BF16	~64 GB	A100 80 GB	Watch KV headroom at long context

FP8 uses H100/Blackwell Tensor Cores. On A100, prefer --quantize bitsandbytes, GPTQ, or AWQ instead of FP8.

Context length vs concurrency

TGI v3 zero-config mode auto-tunes batch and token limits to fill available VRAM. The tradeoff is explicit in Hugging Face’s TGI docs : a model with 128K context on a GPU that can only hold three full-context slots serves three concurrent requests at max context.

Goal	Tuning approach
Max concurrent users	Lower `--max-input-length` (e.g. 64K → 32K doubles slots on same GPU)
Max context per request	Accept lower concurrency; scale replicas horizontally
Predictable latency	Cap `--max-batch-prefill-tokens` and queue depth alerts


docker run --gpus all --shm-size 1g -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct \
  --max-input-length 8192 \
  --max-total-tokens 12288

Leave limits undefined only when you want TGI to maximize hardware utilization on startup — fine for homogenous prod, risky for shared dev clusters.

Quantization on TGI

Method	When to use	GPU requirement
None (BF16)	Quality-sensitive routes, enough VRAM	Any datacenter GPU
FP8	H100-class throughput	H100, not A100
bitsandbytes	INT8 on A100 when FP8 unavailable	CUDA
GPTQ / AWQ	Pre-quantized Hub weights, smallest footprint	Check model card

Quantization reduces weight VRAM; KV cache still scales with context and concurrency. A quantized 70B that fits in 40 GB weights can still OOM under long concurrent sessions.

Shared memory (`--shm-size`)

TGI uses CUDA IPC between processes. Without adequate shared memory, the server may accept the first requests then fail:


docker run --gpus all --shm-size 1g ...

Increase to 2g or 4g for 70B+ multi-process layouts. On Kubernetes, mount an emptyDir with medium: Memory or raise container shm — see Kubernetes.

CPU-only and edge

TGI targets GPU inference. CPU mode exists (--disable-custom-kernels) but throughput is not competitive with llama.cpp on the same hardware. Use llama.cpp or Ollama for laptop/CPU edge; use TGI when you have datacenter GPUs and Hub-native serving.

Hardware by workload

Workload profile	Typical GPU	TGI config hint
Internal API, 7B–8B, <50 QPS	1× L40S / A10G	BF16 8B, moderate context
Multi-tenant chat, 7B–13B	1× A100 80 GB	Quantized 13B or BF16 8B + high batch
70B quality tier	1–2× H100 80 GB	FP8 or 2-shard BF16
Batch overnight jobs	Spot A100/H100 pool	Fixed `max_tokens`, no streaming

Compare dollar-per-token against cloud APIs on TGI cost before buying hardware.

Troubleshooting OOM

Symptom	Fix
OOM on first request	Model too large for GPU — quantize or `--num-shard` across GPUs
OOM under load	Lower `--max-input-length` or reduce concurrent requests
Intermittent CUDA IPC errors	Raise `--shm-size`
OOM after Hub model upgrade	New weights larger — re-benchmark VRAM table above

TGI overview — when TGI beats Ollama and vLLM
TGI server — Docker flags and first request
TGI cost — GPU hourly rates vs API tokens
llama.cpp — fit models on laptop RAM
vLLM — alternative engine if TGI maintenance status matters