Skip to Content
Self-hostingTGIAPI

TGI API

The TGI API exposes an OpenAI-compatible HTTP surface. Point any OpenAI SDK, LangChain adapter, or custom client at the TGI server base URL — no provider-specific rewrites.

Base URL

DeploymentBase URL
Local Docker (-p 8080:80)http://localhost:8080/v1
K8s ingresshttps://tgi.your-domain.com/v1
HF Inference EndpointsProvided in the endpoint dashboard

The API key can be any non-empty string when TGI runs without auth — set a real key at your ingress layer in production.

Chat completions

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer tgi" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Classify intent: refund request"}],
    "max_tokens": 64,
    "temperature": 0
  }'

Request fields

FieldRequiredNotes
modelYesMust match the --model-id loaded by the server
messagesYesStandard OpenAI chat message array
max_tokensYesCap output length — enforce in every client
temperatureNoLower for classification; higher for creative tasks
streamNoSet true for SSE token streaming

Streaming

const stream = await client.chat.completions.create({
  model: "meta-llama/Llama-3.1-8B-Instruct",
  messages: [{ role: "user", content: "Draft a one-line status update." }],
  max_tokens: 128,
  stream: true,
});
 
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

Streaming reduces time-to-first-token for chat UIs. Attribute spans with inference.stream=true for latency dashboards.

OpenAI SDK

import OpenAI from "openai";
 
const client = new OpenAI({
  baseURL: process.env.TGI_BASE_URL ?? "http://localhost:8080/v1",
  apiKey: process.env.TGI_API_KEY ?? "tgi",
});
 
const response = await client.chat.completions.create({
  model: "meta-llama/Llama-3.1-8B-Instruct",
  messages: [{ role: "user", content: "Extract entities from: ACME Corp, Q3 renewal" }],
  max_tokens: 128,
});

Works with Python (openai package), Node, and any HTTP client that speaks the OpenAI schema.

Server-enforced limits

TGI rejects requests that exceed server flags regardless of client max_tokens:

Server flagAPI effect
--max-input-length4xx when prompt tokens exceed cap
--max-total-tokens4xx when input + requested output exceeds cap

Configure limits on the server and validate in CI that agents always send max_tokens.

Health and info endpoints

EndpointPurpose
GET /healthLiveness — returns 200 when the server is ready
GET /infoModel metadata, version, and loaded weights info

Use /health for K8s probes. See Kubernetes for probe configuration.

Version differences

Older TGI builds may expose /generate instead of /v1/chat/completions. Pin your container image tag in production and test the path your client expects.

SymptomFix
404 on /v1/chat/completionsUpgrade TGI or switch to the legacy generate endpoint
422 on large promptsLower prompt size or raise --max-input-length
Empty stream chunksCheck stream_options compatibility with your SDK version

Instrumentation

Tag every API call with span labels for cost attribution:

// Pseudocode — wrap your OpenAI client middleware
span.setAttribute("inference.backend", "tgi");
span.setAttribute("model_id", "meta-llama/Llama-3.1-8B-Instruct");
span.setAttribute("max_tokens", 64);

See Observability for Prometheus metrics on the server side.

Last updated on
ServerKubernetes