llama.cpp memory and GGUF

llama.cpp memory usage is weights (GGUF quant) plus KV cache (context length × batch). Queries like llama cpp fit target, llama.cpp quantization, and what is llamacpp often land here — GGUF format and --fit-target are the levers that keep edge inference usable.

Hub: llama.cpp.

GGUF quantization

Models ship as GGUF files with embedded quantization:

Download from Hugging Face (TheBloke, bartowski, or official repos) or pull via Ollama which uses llama.cpp under the hood.

llama cpp and llama.cpp refer to the same runtime — GGUF is the model format it expects.

Context and RAM

Rule: larger -c without more RAM increases swap thrashing on laptops. Cap per task — do not default to 128K for agent loops.

Datacenter VRAM tables: TGI hardware — same weights-vs-KV-cache tradeoff at different scale.

--fit and --fit-target

llama-server can auto-tune unset memory args to fit your GPU or unified memory:

Example: 24 GB RTX 4090 with 8B Q5 — leave headroom for desktop GPU workloads:

./llama-server \
  -m models/llama-3.2-8b-instruct.Q5_K_M.gguf \
  --host 0.0.0.0 --port 8080 \
  -ngl 99 \
  --fit on \
  --fit-target 2048

If you set -c explicitly, --fit respects your cap when possible. Lower --fit-target when other GPU workloads run alongside the server.

Sizing checklist

1. Pick quant — start Q4_K_M, bump to Q5_K_M for code if RAM allows
2. Set -n (max output tokens) on every call
3. Set -c to task need (4K–8K for most agent turns)
4. Use --fit-target on shared GPUs
5. Benchmark quality on your repo — do not assume Q2 is fine for code

Related

• llama.cpp hub
• Server mode
• Apple Silicon
• Context hygiene