llama.cpp on Apple Silicon
llama.cpp Apple Silicon support via the Metal backend makes it the default local inference stack for Mac developers. Unified memory means weights and KV cache compete with the OS — sizing differs from discrete-GPU Linux boxes.
Hub: llama.cpp. Memory and quant details: Memory & GGUF.
Why Metal matters
| Property | Apple Silicon + llama.cpp | Discrete GPU Linux |
|---|---|---|
| Memory | Unified — shared with OS | Dedicated VRAM |
| Backend | Metal (-ngl 99 typical) | CUDA |
| Dependencies | Minimal — no NVIDIA stack | CUDA drivers required |
| Best fit | Solo dev, air-gapped laptop | Datacenter batching |
RAM-based model picks
| Mac RAM | Practical model range | Notes |
|---|---|---|
| 8 GB | 3B–7B Q4 only | Tight — watch memory pressure |
| 16 GB | 7B–8B Q4; 13B Q4 marginal | Default dev Mac tier |
| 32 GB+ | 13B–14B Q4/Q5; 34B Q4 experimental | Agent loops viable |
Monitor Memory Pressure in Activity Monitor during sustained agent sessions — swap kills interactive latency.
Server on Mac
./llama-server \
-m models/llama-3.2-8b-instruct.Q5_K_M.gguf \
--host 127.0.0.1 --port 8080 \
-ngl 99 \
-c 8192 \
--fit on \
--fit-target 2048Bind 127.0.0.1 unless you explicitly need LAN access. Full server guide: Server mode.
Use --fit-target to leave headroom for Xcode, browsers, and IDE indexing — unified memory is not “free VRAM.”
Thermal and session caps
M-series chips throttle under sustained load:
- Long agent loops heat the chassis — latency spikes after 10–20 minutes
- Set session caps in your IDE (Cursor, OpenCode)
- Prefer shorter context (
-c) to reduce memory bandwidth load - Pause between heavy batch jobs
vs Ollama on Mac
Ollama wraps llama.cpp with ollama pull ergonomics. Go direct to llama.cpp when you need explicit GGUF paths, --fit-target control, or minimal dependencies without Ollama’s daemon.