TGI guardrails – Tokenminning

Guardrails on TGI follow the same progressive enforcement model as other inference backends: instrument first, alert second, block last.

Progressive tiers

Tier	Enforcement	TGI-specific implementation
Soft	Alerts only	Pager on queue depth and P95 TTFT from Observability
Medium	Runtime limits	Per-tenant rate limits at ingress; `max_tokens` required in API schema
Hard	CI + server block	`--max-total-tokens` enforced; CI fails on agents missing output caps

Start at soft. Move to medium when you have two weeks of baseline metrics. Move to hard when a single runaway agent can exhaust GPU memory.

Server-side caps

Configure on the TGI server:


--max-input-length 4096 \
--max-total-tokens 8192

TGI rejects requests that exceed these limits regardless of client behavior. This is your last line of defense — not your only control.

Client-side requirements

Every call through the TGI API must include max_tokens:


await client.chat.completions.create({
  model: "meta-llama/Llama-3.1-8B-Instruct",
  messages: messages,
  max_tokens: 256, // required — never omit
});

Add a lint rule in CI that fails when max_tokens is missing from agent tool definitions.

Ingress rate limits

TGI does not ship per-tenant isolation. Enforce at the gateway:

Control	Example
Requests per minute	60 RPM per API key
Concurrent connections	10 per tenant
Payload size	Reject prompts > 32 KB at ingress

Pair rate limits with span tags (feature, session_id) so alerts name the route, not just the pod.

Anti-patterns

Anti-pattern	Why it fails
`max-input-length` at model maximum for all routes	KV cache exhaustion; terrible batching
Soft alerts with no owner	Queue depth spikes for weeks before anyone acts
Client-only caps, no server flags	One buggy agent bypasses everything
One deployment for unrelated model tiers	Cannot cap or scale tiers independently

Spike alerts

Page when a feature’s token burn hits 4× week-over-week. Slice by feature and model_id span tags — not by invoice line items you get weeks later.

Observability — metrics that feed soft-tier alerts
API — request fields agents must send
Progressive guardrails — framework-wide enforcement model