llama.cpp server mode

llama-server (also called llama.cpp server mode) exposes an OpenAI-compatible HTTP API so IDEs, scripts, and agents share one local model process instead of spawning CLI per request.

Hub: llama.cpp. Queries like llama cpp server and llamacpp server usually mean this surface.

Start the server

./llama-server \
  -m models/llama-3.2-8b-instruct.Q4_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -c 8192 \
  -ngl 99

Let --fit auto-size unset memory args: Memory guide.

OpenAI client wiring

import OpenAI from "openai";
 
const client = new OpenAI({
  baseURL: "http://localhost:8080/v1",
  apiKey: "llama",
});

Works with Cline, Aider, OpenCode BYOK, and custom agents — set baseURL to your server and pick a model name the server reports.

When to use server vs CLI

Server mode amortizes model load time and keeps weights hot in VRAM/unified memory.

Metering

llama.cpp does not ship Prometheus metrics. Instrument at the client or proxy:

Tag spans with inference.backend=llama-cpp for fair comparison against Ollama and cloud APIs.

Guardrails

• Set max_tokens on every API call — server does not cap output by default
• Reverse-proxy rate limits when exposing on LAN
• Auth or VPN — never bind 0.0.0.0 without access control on shared networks
• One model per process; scale with multiple server instances behind a router

Related

• llama.cpp hub
• Memory & GGUF
• Apple Silicon
• Local inference