TGI server – Tokenminning

The TGI server is a single container (or K8s pod) that loads a Hub model and serves inference over HTTP. This page covers local and single-node deployment; for production clusters see Kubernetes.

Docker quick start


docker run --gpus all --shm-size 1g -p 8080:80 \
  --mount type=bind,src=$PWD/data,dst=/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct \
  --max-input-length 4096 \
  --max-total-tokens 8192

Flag	Purpose
`--gpus all`	Expose GPU to the container
`--shm-size 1g`	Shared memory for tensor ops — increase for large models
`-p 8080:80`	Map host port 8080 to TGI’s HTTP port
`--mount type=bind,src=$PWD/data,dst=/data`	Persist HF cache between restarts

Verify the server is up:


curl http://localhost:8080/health

Model selection

TGI loads directly from Hub model-id:


--model-id mistralai/Mistral-7B-Instruct-v0.3

Consider:

License — Llama, Mistral, and others have use restrictions
Gated models — require HF account approval and HF_TOKEN
Custom modeling code — trust-remote-code when required (audit the code first)

Match model tier to task per Model routing — do not serve 70B when 8B passes your eval.

Memory and throughput flags

Parameter	Effect
`--max-input-length`	Caps prompt size — protects KV memory
`--max-total-tokens`	Input + output ceiling per request
`--max-batch-prefill-tokens`	Prefill batch budget
`--quantize`	AWQ/GPTQ paths when supported

Required: enforce max_tokens at the client and server. TGI rejects oversize requests when limits are configured.

Gated models

Export your Hugging Face token before starting the server:


export HF_TOKEN=hf_...
docker run --gpus all --shm-size 1g -p 8080:80 \
  -e HF_TOKEN \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct

Rotate tokens on a schedule — stale tokens break CI pulls without obvious errors. See Support for enterprise token management.

First request

Once the server is running, send a request via the API:


import OpenAI from "openai";
 
const client = new OpenAI({
  baseURL: "http://localhost:8080/v1",
  apiKey: "tgi",
});
 
const response = await client.chat.completions.create({
  model: "meta-llama/Llama-3.1-8B-Instruct",
  messages: [{ role: "user", content: "Summarize this ticket in one sentence." }],
  max_tokens: 64,
});

Troubleshooting

Model download slow on start — mount a persistent volume for the HF cache or pre-download weights.

OOM after deploy — lower --max-input-length or enable --quantize.

Container exits immediately — check GPU driver compatibility and increase --shm-size.

OpenAI client 404 — confirm the chat completions path on your TGI host (/v1/chat/completions) and TGI version; older builds differ.

API — endpoint reference and client wiring
Kubernetes — production multi-replica deploys
Observability — metrics once the server is live