TGI observability
Instrument TGI like any production inference backend: server-side Prometheus metrics plus client-side span tags. Without both, you see GPU utilization but not cost per feature.
Prometheus metrics
TGI exposes Prometheus metrics at http://localhost:8080/metrics on the inference host. Scrape it from your Prometheus server or agent running in the same cluster.
| Metric area | What it tells you | Action when bad |
|---|---|---|
| Queue depth | Requests waiting for batch slots | Scale replicas or reduce max-input-length |
| Time to first token (TTFT) | Batch contention or prefill bottleneck | Dedicated model pools per workload |
| GPU memory | KV cache pressure | Enable quantization or split across GPUs |
| Request duration | End-to-end latency | Check ingress timeouts and client max_tokens |
Alert thresholds (starting points)
| Alert | Condition | Severity |
|---|---|---|
| Queue depth high | Sustained > 10 for 5 min | Warning |
| TTFT P95 spike | > 2× baseline for 10 min | Warning |
| GPU OOM restarts | Any pod restart with OOMKilled | Critical |
| Error rate | 5xx > 1% for 5 min | Critical |
Tune thresholds per model and traffic profile after one week of baseline data.
Span tags
Attribute every client request with labels that slice dashboards by feature:
inference.backend=tgi
model_id=meta-llama/Llama-3.1-8B-Instruct
feature=support-bot
session_id=<uuid>
max_tokens=64| Tag | Purpose |
|---|---|
inference.backend | Filter TGI vs vLLM vs cloud API in one dashboard |
model_id | Cost attribution per model tier |
feature | Cost per outcome — which product surface burned tokens |
session_id | Debug multi-turn agent loops |
max_tokens | Verify guardrails are actually set |
Emit spans from your OpenAI SDK middleware, not from TGI itself — TGI metrics cover server health; spans cover business context.
Dashboards
Build three panels minimum:
- Throughput — requests/sec, tokens/sec (input + output)
- Latency — TTFT P50/P95, total duration P95
- Efficiency — GPU utilization vs tokens/sec (identify idle GPUs)
Compare TGI against vLLM on the same dashboard during migration — normalized by model and max_tokens.
Log correlation
Pair Prometheus alerts with structured logs from your API gateway:
{
"request_id": "req_abc123",
"model_id": "meta-llama/Llama-3.1-8B-Instruct",
"prompt_tokens": 412,
"completion_tokens": 58,
"latency_ms": 340,
"inference.backend": "tgi"
}When queue depth spikes, correlate with feature tags to find the noisy neighbor route.
Quality vs vLLM
If quality drops below vLLM on the same hardware, check quantization and batch settings before blaming the engine. Re-benchmark with matched configs and log model_id + quant flags in spans.
Related
- Guardrails — alerts that escalate to hard caps
- API — client instrumentation at the SDK layer
- Kubernetes — scrape config inside the cluster