vLLM: self-host models with GPU serving
vLLM serves open-weight models with continuous batching and an OpenAI-compatible API. Use it when you manage the GPU host and want direct control over model loading, context limits, parallelism, and caching. See the official documentation for current platform and model support.
When to choose vLLM
| Signal | What it means |
|---|---|
| You own or rent a GPU host | vLLM runs on hardware you provision — unlike iwant, which also rents the machine |
| Steady API or batch traffic | Continuous batching pays off when utilization stays high |
| You need engine flags | Prefix caching, tensor parallelism, and quantization are first-class configuration |
Use Ollama for solo dev; vLLM when Ollama stops scaling on concurrency or GPU utilization.
Setup and operation
| Guide | What you’ll learn |
|---|---|
| Serve a model | vllm serve, OpenAI client, architecture |
| Size GPU memory | VRAM tables, throughput knobs, quantization |
| Enable prefix caching | --enable-prefix-caching, stable prefixes |
Expected impact
| Scenario | Outcome |
|---|---|
| High QPS, >60% GPU util | 70–95% vs cloud API |
| Low QPS, idle GPU | Negative |
| Sporadic dev experiments | Use Ollama instead |
Figures are illustrative ranges — meter your own utilization and cost per outcome (self-hosting cost model).
Model-specific deployment guides
Each guide links the actual upstream configuration, explains hardware and checkpoint variants, and includes a first API request. Browse all documented models for other deployment options.
vLLM alternatives
| Tool | Choose when |
|---|---|
| iwant | You want a documented model name to become a GCP endpoint without assembling VMs, engine flags, and teardown yourself. |
| SGLang | You need RadixAttention prefix reuse, structured generation, or model-specific parsers on hardware you configure. |
| TGI | You already run Hugging Face Text Generation Inference on Kubernetes and the checkpoint is on the supported-model list. |
| Ollama | You want local ollama pull ergonomics, IDE chat, and low-ops serving on a single machine. |
| llama.cpp | You serve GGUF weights on CPU, Apple Silicon, or a modest GPU with minimal dependencies. |
| LocalAI | You need one OpenAI-compatible URL over multiple backends or gallery configurations. |
Deeper comparison: Alternatives to vLLM. All launch methods: self-hosting hub.
Related reading
- Self-hosting hub — economics, model directory, and launch comparison
- Text generation inference — runtime terminology and workload fit
- Local inference practice — when to self-host and how to meter GPU time
- Alternatives to vLLM
- Prompt caching
- Zed local models
Last updated on