SGLang: self-host models with OpenAI-compatible APIs
SGLang serves open models on GPUs with continuous batching, prefix reuse through RadixAttention, and structured generation. Begin with a model-specific configuration: hardware, weight format, reasoning parsers, and tool-call parsers can differ between checkpoints. See the official documentation for current platform and model support.
When to choose SGLang
| Signal | What it means |
|---|---|
| Prefix-heavy agent sessions | RadixAttention reuses shared prompt prefixes across requests |
| Structured or tool output | Model-specific parsers are part of the serving configuration |
| You benchmark against vLLM | Winners vary by model family and concurrency — measure on your hardware |
Pair with vLLM benchmarks on identical models before committing production traffic.
Setup and operation
| Guide | What you’ll learn |
|---|---|
| Install and serve a model | CUDA install, sglang serve, parsers, first request |
Compare tokens vs GPU time on the self-hosting cost model before renting large multi-GPU hosts.
Model-specific deployment guides
Each guide links the actual upstream configuration, explains hardware and checkpoint variants, and includes a first API request. Browse all documented models for other deployment options.
SGLang alternatives
| Tool | Choose when |
|---|---|
| iwant | You want a documented model name to become a GCP endpoint without assembling VMs, engine flags, and teardown yourself. |
| vLLM | You manage the GPU host and need continuous batching, prefix caching, and direct control over vllm serve settings. |
| TGI | You already run Hugging Face Text Generation Inference on Kubernetes and the checkpoint is on the supported-model list. |
| Ollama | You want local ollama pull ergonomics, IDE chat, and low-ops serving on a single machine. |
| llama.cpp | You serve GGUF weights on CPU, Apple Silicon, or a modest GPU with minimal dependencies. |
| LocalAI | You need one OpenAI-compatible URL over multiple backends or gallery configurations. |
All launch methods: self-hosting hub.
Related reading
- Self-hosting hub — economics, model directory, and launch comparison
- Text generation inference — runtime terminology and workload fit
- Local inference practice — when to self-host and how to meter GPU time
Last updated on