TGI on Kubernetes – Tokenminning

Production Text Generation Inference runs as a Deployment behind a Service and Ingress. This page covers the patterns that keep TGI stable under load — not a copy-paste Helm chart, but the decisions that matter for token observability and uptime.

Deployment shape


Ingress (TLS, rate limits)
        ↓
   TGI Service (ClusterIP)
        ↓
   TGI Deployment (N replicas)
        ↓
   GPU node pool (node selector)

Each replica runs one model instance with continuous batching. Scale replicas horizontally when queue depth stays high; scale vertically (bigger GPU) when a single model does not fit in VRAM.

Model weights

Pattern	When to use
Init container + HF Hub pull	Default — fresh weights on every deploy
PVC with cached weights	Large models where Hub pull time blocks rollouts
Pre-baked image	Air-gapped or regulated environments

For gated models, mount HF_TOKEN from a Kubernetes Secret:


env:
  - name: HF_TOKEN
    valueFrom:
      secretKeyRef:
        name: hf-token
        key: token

Never bake tokens into container images. Rotate secrets on the same schedule as your CI tokens.

Resource requests


resources:
  limits:
    nvidia.com/gpu: 1
  requests:
    nvidia.com/gpu: 1
    memory: "32Gi"

Concern	Pattern
GPU scheduling	`nodeSelector` or `tolerations` for GPU taints
Shared memory	Set `emptyDir` with `medium: Memory` or increase container `shm`
Probes	`livenessProbe` and `readinessProbe` on `GET /health`

Undersized memory requests cause OOM kills mid-batch. Size from model weights + expected concurrent batch depth.

Scaling

Signal	Action
Sustained queue depth	Add replicas via HPA
GPU utilization < 40%	Reduce replicas or consolidate models
P95 time-to-first-token rising	Check batch contention before scaling

HPA on custom metrics (queue depth, GPU utilization) beats CPU-based scaling for inference workloads.

Ingress and multi-tenancy

Put rate limits and auth at the ingress layer — TGI itself does not ship tenant isolation:

Layer	Responsibility
Ingress	TLS, per-tenant rate limits, API keys
TGI Service	Load balance across replicas
TGI pod	Model inference, token limits

See Guardrails for progressive enforcement tiers.

Upgrades

Model swaps are deploy events — treat them like application releases:

Deploy new replica set with updated --model-id or image tag
Wait for /health on new pods
Shift traffic (blue/green or rolling update)
Drain old replicas

Pre-warm weights on PVC to avoid multi-minute rollout stalls on 70B+ models.

Managed alternative

Hugging Face Inference Endpoints runs managed TGI on AWS, GCP, and Azure. Meter managed endpoints like cloud APIs; self-managed K8s TGI meters GPU-hours. See Support for when managed beats self-hosted.

Troubleshooting

Pods stuck Pending — no GPU nodes available or missing nvidia.com/gpu resource.

CrashLoop on start — model too large for GPU; enable quantization or add tensor parallelism.

Intermittent 503 — readiness probe fires before model load completes; increase initialDelaySeconds.

Cross-AZ latency — co-locate clients and TGI in the same region/AZ.

Server — Docker flags that map directly to K8s args
API — endpoint paths for ingress routing
Observability — Prometheus scrape config for TGI pods

Last updated on June 24, 2026

Cost Observability