vLLM GPU sizing and tuning

Size GPUs for weights + KV cache headroom. Undersized → OOM; oversized → idle depreciation.

Hub: vLLM.

VRAM by model size


vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --quantization awq \
  --dtype auto

Benchmark quantized vs FP16 on your eval set before cutting cloud traffic.

Flag	Effect
`--max-num-seqs`	Concurrency — raise until latency breaks SLO
`--gpu-memory-utilization`	VRAM fraction for KV (default 0.9)
`--max-model-len`	Cap context — lowers KV memory
`--enable-prefix-caching`	Prefix KV reuse — guide

Prohibited: max-model-len at 128K for every request.

Prometheus at http://localhost:8000/metrics:

Metric	Use
`vllm:gpu_cache_usage_perc`	KV pressure
`vllm:num_requests_running`	Concurrency vs SLO
`vllm:time_to_first_token_seconds`	Streaming TTFT

Tag requests with inference.backend=vllm and feature for cost per outcome.