llama.cpp: serve GGUF models on CPU and GPU
llama.cpp runs GGUF models on CPUs and supported GPUs, including Apple Silicon. Use its command-line tools for local inference or llama-server for an OpenAI-compatible endpoint. Choose a compatible conversion and leave memory for context and concurrent requests. See the official documentation for current platform and model support.
When to choose llama.cpp
| Signal | What it means |
|---|---|
| GGUF weights | Quantized checkpoints tuned for llama.cpp memory profiles |
| CPU, edge, or Apple Silicon | Runs where CUDA serving stacks do not |
| Minimal dependencies | Single binary workflow without a full Python serving stack |
Use vLLM or SGLang when you need multi-tenant GPU batching at datacenter scale.
Setup and operation
| Guide | What you’ll learn |
|---|---|
| Start llama-server | OpenAI-compatible server mode |
| Size memory and GPU offload | Context, layers, and concurrency |
| Run on Apple Silicon | Unified memory and Metal offload |
Model-specific deployment guides
Each guide links the actual upstream configuration, explains hardware and checkpoint variants, and includes a first API request. Browse all documented models for other deployment options.
llama.cpp alternatives
| Tool | Choose when |
|---|---|
| iwant | You want a documented model name to become a GCP endpoint without assembling VMs, engine flags, and teardown yourself. |
| vLLM | You manage the GPU host and need continuous batching, prefix caching, and direct control over vllm serve settings. |
| SGLang | You need RadixAttention prefix reuse, structured generation, or model-specific parsers on hardware you configure. |
| TGI | You already run Hugging Face Text Generation Inference on Kubernetes and the checkpoint is on the supported-model list. |
| Ollama | You want local ollama pull ergonomics, IDE chat, and low-ops serving on a single machine. |
| LocalAI | You need one OpenAI-compatible URL over multiple backends or gallery configurations. |
All launch methods: self-hosting hub.
Related reading
- Self-hosting hub — economics, model directory, and launch comparison
- Text generation inference — runtime terminology and workload fit
- Local inference practice — when to self-host and how to meter GPU time
Last updated on