TGI cost per token – Tokenminning

Self-hosted Text Generation Inference does not bill per token on an invoice — you pay GPU-hours, power, and engineer time. To compare fairly against cloud APIs, convert both sides to cost per outcome or cost per million tokens at your actual utilization.

This page answers the queries behind high-impression TGI landing pages: “text generation inference cost,” “TGI vs OpenAI pricing,” and “when does self-hosting break even.”

Two cost models

Model	What you pay	Risk
Cloud API	Input + output tokens × list price	Spiky traffic; frontier model drift
TGI on GPU	GPU $/hr × wall-clock utilization	Idle GPUs; ops incidents; upgrade churn

Cloud APIs win at low volume and high model churn. TGI wins when QPS is steady, model choice is stable, and GPU utilization stays high.

Worked example: 8B instruct tier

Assume Llama 3.1 8B on one A100 80 GB at $2.50/hr (typical cloud GPU rental — adjust to your contract).

Rough throughput for chat at batch size 8: ~3,000–6,000 output tokens/minute depending on context length and batching (measure on your stack — do not trust generic benchmarks).

Metric	Calculation	Result (illustrative)
Output tokens per hour	4,000 tok/min × 60	~240M tok/hr at peak
Cost per 1M output tokens	$2.50 / 240	~$0.01/M out
At 50% GPU utilization	Double effective $/M	~$0.02/M out

Compare to cloud list (check current rates on Narev or provider dashboards):

Provider / model (indicative)	Output $/M tokens
GPT-4o mini	~$0.60
Claude Haiku 3.5	~$1.25
Llama 3.1 8B on major APIs	~$0.08–0.20

At high utilization, TGI on rented A100 often beats frontier APIs by an order of magnitude on output tokens. At 10% utilization (one dev GPU running two hours a day), the same math yields ~$0.10/M out — cloud mid-tier APIs can be cheaper because you are not amortizing idle silicon.

Break-even checklist

TGI is likely cheaper when all of these are true:

Steady QPS — not just occasional IDE experiments (Ollama fits spiky dev better)
Utilization > 40% on the GPU serving TGI — measure with nvidia-smi and Observability
Stable model — re-tuning VRAM and redeploying every week erases savings
Quality bar met — failed local attempts that escalate to cloud add hidden API cost (Article II)
Ops cost included — on-call, image upgrades, HF token rotation (Support)

If any item fails, run the hybrid pattern: local draft on Ollama, production on TGI or cloud.

Input vs output on self-hosted stacks

Cloud APIs often charge input tokens heavily. TGI cost is mostly wall-clock on GPU — long prompts still cost compute during prefill, but there is no separate input list price.

Pattern	Cloud API impact	TGI impact
Bloated RAG context every turn	High input $	Longer prefill, lower max concurrency
Agent loops (20+ turns)	Input + output stack	Same GPU hour, worse queue depth
Short prompts, long completions	Output-weighted	Favorable for batching

Trim retrieval before inference per Output and RAG. Cap max_tokens in clients and on the TGI server.

Managed TGI vs self-managed

Option	Cost shape	When it wins
HF Inference Endpoints	Per-GPU-hour + HF markup	Fast prod without K8s team
Self-managed TGI on K8s	GPU + cluster + engineer	High scale, custom networking
Cloud API	Per token	Low volume, frontier models

Meter managed endpoints like cloud APIs. Meter self-managed TGI like owned or rented GPU capacity.

Hidden costs teams forget

Cost	Why it matters
Idle GPU	24/7 node that serves 2 hrs/day multiplies $/token
Weight pull time	Hub downloads on every cold start — use PVC cache (K8s)
Failed quant deploy	Trial FP8 on A100 → rollback → engineer time
Quality regression	Cloud escalation after bad local merges
Power	On-prem: add ~$0.10–0.30/kWh × GPU TDP to hourly math

Compare engines fairly

Before standardizing on TGI, benchmark measured $/M tokens at your QPS against vLLM on the same GPU and model. Throughput differences of 20–40% directly change break-even. Hugging Face now positions vLLM and SGLang for many new deployments; TGI remains valid for Hub-native K8s stacks already in production — pick on numbers, not logo preference.

Instrumentation

Tag every client call:


span.setAttribute("inference.backend", "tgi");
span.setAttribute("inference.gpu_hour_rate", 2.5);
span.setAttribute("llm.usage.output_tokens", usage.completion_tokens);

Divide allocated GPU cost by summed output tokens weekly per feature tag. That is your real cost per outcome input for routing decisions.

See Observability for Prometheus metrics (tgi_batch_current_size, queue depth) that explain utilization swings.

TGI overview — architecture and guides
TGI hardware — VRAM sizing before you rent GPUs
Self-hosting — workload fit and decision flow
Local inference — optimization stack step 8
Model routing — hybrid local/cloud escalation