Serve a model with SGLang: setup and first request

Start a local SGLang server and send a bounded chat request before wiring an application to it. This walkthrough uses Qwen/Qwen3-0.6B, a small model that avoids the gated-download step in the previous Llama example. It demonstrates the API workflow, not the hardware requirements of larger models.

Install

Use Linux with a supported NVIDIA GPU and the driver/CUDA environment required by your chosen SGLang release. The current installation guide  specifies CUDA 13 for the current release line; check that guide before installing on an older CUDA system.

Create an isolated Python environment using uv:

pip install uv
uv venv --python 3.12
source .venv/bin/activate
uv pip install --prerelease=allow sglang

The prerelease option follows upstream’s dependency-installation guidance. This command does not pin a release; record the installed version when comparing benchmarks.

Launch and check readiness

In that environment, start the server:

sglang serve Qwen/Qwen3-0.6B \
  --host 127.0.0.1 \
  --port 30000

Wait for model loading to finish. In another terminal on the same host, list the served model:

curl -fsS http://127.0.0.1:30000/v1/models

Then send a request:

curl -fsS http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"What is 2 + 2? Answer briefly."}],"max_tokens":256}'

Check choices[0].message for the generated response and usage for token counts. If the output cap is reached during reasoning, the answer may be incomplete. The official request tutorial  covers API options beyond this first check.

Configure an OpenAI-compatible client

For a JavaScript app with the openai package installed:

import OpenAI from "openai";
 
const client = new OpenAI({
  baseURL: "http://127.0.0.1:30000/v1",
  apiKey: "unused-local-key",
});
 
const completion = await client.chat.completions.create({
  model: "Qwen/Qwen3-0.6B",
  messages: [{ role: "user", content: "What is 2 + 2? Answer briefly." }],
  max_tokens: 256,
});
 
console.log(completion.choices[0]?.message);
console.log(completion.usage);

This server example binds to loopback and does not configure API authentication. The SDK’s placeholder key does not add server authentication. For a remote service, configure the network binding and access controls deliberately using the server arguments reference .

Diagnose failures before scaling

If installation fails, check the release’s driver, CUDA, and Python requirements. If the connection is refused, read startup logs: the server may still be downloading weights or may have exited. A model lookup error calls for checking /v1/models against the client’s model field.

For a larger checkpoint or multi-GPU deployment, use the model-specific upstream configuration rather than extrapolating from this small example. Compare memory use, latency, and concurrency against vLLM on the same workload.

Stop the server and account for GPU time

Ctrl+C in the server terminal stops this foreground local process. If you started it on a rented VM, stopping SGLang does not delete the VM; remove it through your cloud deployment workflow when finished.

Log prompt and completion tokens, GPU runtime, and utilization as described in Local inference. Tag requests with inference.backend=sglang for token metering.