Skip to Content
Self-hostingvLLMServe & tutorial

vLLM serve tutorial

vLLM tutorial path: install, run vllm serve, point OpenAI SDK at http://localhost:8000/v1.

Hub: vLLM.

Install and serve

pip install vllm
 
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 8192

Architecture

Client → vLLM OpenAI server (:8000)
              ↓
    PagedAttention + continuous batching
              ↓
         GPU (CUDA / ROCm)
  • Continuous batching — new requests join in-flight batches
  • PagedAttention — KV cache in non-contiguous pages
  • Tensor parallelism — multi-GPU for large models

OpenAI client

import OpenAI from "openai";
 
const client = new OpenAI({
  baseURL: "http://localhost:8000/v1",
  apiKey: "vllm",
});
 
const response = await client.chat.completions.create({
  model: "meta-llama/Llama-3.1-8B-Instruct",
  messages: [{ role: "user", content: "Classify intent: refund request" }],
  max_tokens: 64,
});

Production patterns

  • Reverse proxy for TLS, auth, rate limits
  • Kubernetes Helm chart + HPA on GPU metrics
  • Separate deployments per model tier — Model routing at URL level
Last updated on
vLLMPrefix caching