Self-host LLMs: choose a model and launch tool
Self-hosting means running open-weight models on hardware you control instead of paying per-token API rates. The trade is fixed GPU cost and ops work for predictable unit economics at high volume — not a free lunch.
Self-hosting is the last lever in the optimization stack. Start with Local inference, then pick a model and launch path below. Evaluating a stack swap? See Alternatives for “alternative to X” comparisons ranked by GPU economics and workload fit.
Start with the model you want to run, then choose how to launch it. For a documented model on cloud GPUs, iwant handles provisioning and serving behind iwant up MODEL. For an existing GPU host, vLLM and SGLang let you configure the serving engine directly. Ollama and llama.cpp cover local use; LocalAI adds a shared API over multiple backends.
When self-hosting wins
| Signal | What it means |
|---|---|
| Steady high QPS on a narrow task mix | Fixed GPU amortizes faster than API tokens |
| Data cannot leave your network | Compliance or IP constraints block cloud APIs |
| Predictable batch windows | Overnight jobs, eval pipelines, internal tools |
| You already own GPU capacity | Idle cluster time is cheaper than marginal API spend |
When cloud APIs still win
| Signal | What it means |
|---|---|
| Spiky or exploratory traffic | Pay-per-token absorbs variance; GPUs sit idle |
| You need frontier quality | Open weights lag closed models on hard reasoning |
| Small team, no inference SRE | Serving, upgrades, and incident response are real work |
| Low monthly inference spend | Hardware depreciation exceeds API bills |
Cost model: tokens vs GPU time
Cloud APIs bill input + output tokens. Self-hosted stacks bill GPU-seconds, VRAM headroom, and engineer time.
Compare fairly:
- Normalize to cost per outcome — resolved tickets, classified rows, merged PRs — not raw token volume.
- Meter wall-clock and GPU utilization — a 7B model at 30% GPU load is not “free” because there is no API invoice.
- Include idle capacity — a dedicated dev GPU that runs Ollama two hours a day still depreciates 24/7 unless shared.
- Add routing overhead — hybrid stacks (local for draft, cloud for commit) need explicit escalation logs per Article II.
Narev helps compare cloud token rates when you benchmark local quality against API baselines. Runtime workload tables and guardrail tiers: Text generation inference.
Find your model
Each guide covers the exact checkpoint, hardware for its launch paths, first API request, troubleshooting, and cleanup. GPU counts below describe iwant’s cloud configuration; other tools can use different layouts.
- Self-host gpt-oss-20b: iwant on one L4, direct GPU serving, and local deployment options.
- Self-host DeepSeek-V4-Flash: iwant on eight H100s and engine-specific serving settings.
- Self-host DeepSeek-V4.1-Flash: iwant on eight H200s, host-memory offload, and text-only versus multimodal setup.
- Self-host Ling-3.0-flash FP8: iwant on eight H200s and smaller direct-runtime layouts.
- Self-host MiMo-V2.6-Flash-RL: iwant on eight H200s, weight-format support, and checkpoint differences.
- Self-host Step-3.7-Flash: iwant on eight H200s, BF16 serving, and model aliases.
Compare ways to launch
The useful comparison is how much deployment work a tool handles for you. A command that starts a server on an existing machine and a command that rents a GPU and starts the server have different prerequisites.
| Tool | Start from | What it handles | What you still arrange |
|---|---|---|---|
| iwant | Model name and configured cloud account | GPU provisioning, model setup, serving flags, readiness check, endpoint key, teardown command | GCP quota, capacity, budget, and application validation |
| vLLM | Compatible GPU host and checkpoint | Model loading, batching, and serving API | Host provisioning, runtime setup, model flags, access controls, resource cleanup |
| SGLang | Compatible host and model configuration | Model serving, prefix reuse, and structured generation | Host provisioning, installation, model configuration, and cloud cleanup |
| TGI | Host/container infrastructure and supported model | Containerized inference and serving API | Infrastructure lifecycle and supported-architecture checks |
| Ollama | Local machine and library model | Model download and local serving | Hardware capacity and any remote deployment lifecycle |
| llama.cpp | Host and compatible GGUF weights | Inference and optional server | Build/install, checkpoint format, hardware fit, and host lifecycle |
| LocalAI | Host and backend/model configuration | Model gallery setup and a common API across backends | Infrastructure, backend requirements, and cloud cleanup |
For model-name-to-cloud-endpoint deployment, iwant combines these stages in one CLI. You do not need to select the underlying serving configuration. Its upstream workflow currently supports GCP and runs vLLM for the documented models. This automation comparison does not establish a tokens-per-second or startup-time winner.
Choose a direct runtime when you need another hardware topology, checkpoint variant, or serving configuration. Read the model guide first: a runtime’s general model coverage does not mean every checkpoint, precision, or parser combination works.
Decide whether a dedicated GPU makes sense
Self-hosting replaces per-token API billing with hardware runtime and operations work. Compare cost per useful outcome, including idle time, storage, and engineer time. Sporadic requests can leave a rented GPU mostly unused; steady traffic or existing GPU capacity can make the trade more attractive.
Log input and output tokens together with GPU runtime and utilization. Test representative prompts, output quality, and concurrency before changing production traffic. Local inference covers that evaluation, and model routing covers hybrid deployments.
Remove resources after testing
Stopping a local serving process does not remove a cloud VM. With iwant, use iwant down EXACT_CLUSTER_NAME and confirm removal with iwant list. Its idle timer tracks cluster jobs, so a running server can keep resources active without API traffic. Other launch paths require your host or orchestration cleanup workflow.
For engine behavior and terminology, read text generation inference. For stack swaps, use Alternatives to vLLM or Alternatives to iwant.