Skip to Content
Self-hostingModelsgpt-oss-20b

Self-host gpt-oss-20b: GPU requirements and deployment

Run openai/gpt-oss-20b on a cloud GPU with iwant, serve it directly with vLLM or SGLang, or use a local Ollama, llama.cpp, or LocalAI deployment. The model is available in native MXFP4 weights and in a GGUF conversion; choose the format your runtime actually loads.

GPU and storage requirements

iwant requests one L4, at least 8 vCPUs, at least 32 GB of host RAM, and a 200 GB disk. Its server uses an 8,192-token context limit. The official vLLM guide describes a 16 GB model-memory footprint on supported hardware; leave additional headroom for the KV cache and your concurrency. The SGLang example below uses one H100 or H200. Local options use their own model format and memory configuration.

For the raw-runtime layouts and checkpoint variants, consult the linked vLLM and SGLang recipes below. For iwant, preview the selected cloud resources with iwant up gpt-oss-20b --infra gcp --dry-run --yes before spending money.

Deploy with iwant

Complete iwant installation and cloud authentication once. Your GCP project needs billing enabled, GPU quota, and available capacity. Then launch by model name:

iwant up gpt-oss-20b

The CLI guides you through the launch choices and selects the serving configuration automatically. Choose a real launch rather than the dry-run preview when prompted. For a scripted on-demand launch, use iwant up gpt-oss-20b --infra gcp --yes instead. You do not need to choose an image or assemble engine flags.

Save the printed Server URL and API key after startup. Readiness means the models API responded; use the first request below to check inference. GCP is currently supported. For download authentication or provisioning problems, use the shared setup and troubleshooting guide.

Serve with vLLM

Use a compatible GPU host and complete vLLM installation. The official gpt-oss-20b recipe  specifies vLLM 0.10.1 or newer. This example chooses an 8,192-token serving limit and keeps the endpoint local to the host.

Replace YOUR_SERVER_KEY with your chosen server key before starting the process. Run this on the GPU host, with any required HF_TOKEN set in that shell.

export MODEL_API_KEY="YOUR_SERVER_KEY"
 
vllm serve openai/gpt-oss-20b \
  --max-model-len 8192 \
  --host 127.0.0.1 --port 8000 \
  --api-key "$MODEL_API_KEY"

The example binds to loopback. Test from another terminal on the same host, or use an SSH tunnel to reach its port. Stop the foreground process with Ctrl+C when finished; removing a rented host is a separate cloud action.

Serve with SGLang

The official SGLang GPT-OSS cookbook  covers the native openai/gpt-oss-20b checkpoint on one H100 or H200. Complete SGLang installation for your GPU and CUDA environment. The parser names below belong to GPT-OSS, including its reasoning and tool-call response format.

Use your chosen server key and set HF_TOKEN if the download requires authentication. This is an alternative server: stop any earlier process using port 8000 before starting it.

export MODEL_API_KEY="YOUR_SERVER_KEY"
 
sglang serve \
  --model-path openai/gpt-oss-20b \
  --tp 1 \
  --reasoning-parser gpt-oss \
  --tool-call-parser gpt-oss \
  --host 127.0.0.1 --port 8000 \
  --api-key "$MODEL_API_KEY"

Wait for startup, then run the shared API check from the same host or through an SSH tunnel. Ctrl+C stops this foreground server; it does not remove the underlying VM.

Run locally with Ollama

The official Ollama gpt-oss library entry  provides the local 20B model in native MXFP4 format. Complete Ollama setup, check available memory, and run:

ollama run gpt-oss:20b

That downloads the local model and opens a chat session. For the API, use http://127.0.0.1:11434/v1 and model ID gpt-oss:20b. The local server does not require the iwant key. Its library entry describes a 16 GB model-memory footprint; additional context and concurrent requests require more memory.

curl -fsS http://127.0.0.1:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-oss:20b","messages":[{"role":"user","content":"What is 2 + 2? Answer briefly."}],"max_tokens":256}'
 
ollama stop gpt-oss:20b

Stopping the model unloads it; it does not delete a rented machine or uninstall the Ollama service.

Serve the GGUF conversion with llama.cpp

The ggml-org GPT-OSS GGUF repository  converts the OpenAI checkpoint for GGUF runtimes. Use a current llama.cpp installation that supports GPT-OSS. The server reference  documents the download selector, alias, Jinja template, and authentication options used here.

export MODEL_API_KEY="YOUR_SERVER_KEY"
 
llama-server \
  -hf ggml-org/gpt-oss-20b-GGUF \
  --jinja --alias gpt-oss-20b \
  --ctx-size 8192 \
  --host 127.0.0.1 --port 8000 \
  --api-key "$MODEL_API_KEY"

The checkpoint is a GGUF conversion, not the original Hugging Face file layout. Choose GPU offload and context settings for your available memory using the memory guide. The served alias is gpt-oss-20b; use it with the shared request below. Ctrl+C stops the server.

Serve through LocalAI

The LocalAI gallery  includes gpt-oss-20b backed by the ggml-org MXFP4 GGUF conversion. Its Harmony configuration  selects the llama.cpp backend, an 8,192-token context, and Jinja-based message handling.

After LocalAI installation, use the gallery model on a host with enough memory. The CLI reference  covers these launch and API-key options:

export MODEL_API_KEY="YOUR_SERVER_KEY"
 
local-ai run --address 127.0.0.1:8080 \
  --api-keys "$MODEL_API_KEY" gpt-oss-20b

Use http://127.0.0.1:8080/v1, your chosen key, and the ID returned by /models. Gallery configuration and backend availability determine the actual CPU/GPU behavior. Stop the foreground service with Ctrl+C and handle cloud resource cleanup separately.

Verify the model API

After any launch path above, confirm the server with OpenAI-compatible HTTP. Set OPENAI_BASE_URL to the base URL including /v1 (iwant prints this; vLLM and SGLang examples use http://127.0.0.1:8000/v1; Ollama uses port 11434; LocalAI uses 8080). Set OPENAI_API_KEY to the key from iwant or the value you chose when starting the server. Run curls on the GPU host or through SSH when the process binds to loopback.

export OPENAI_BASE_URL="YOUR_SERVER_URL_WITH_V1"
export OPENAI_API_KEY="YOUR_SERVER_API_KEY"

curl -fsS "$OPENAI_BASE_URL/models" \
  -H "Authorization: Bearer $OPENAI_API_KEY"

Copy the served id from the response into SERVED_MODEL_ID.

iwant and vLLM/SGLang often expose openai/gpt-oss-20b; Ollama uses gpt-oss:20b; the llama.cpp example uses gpt-oss-20b. Model aliases can differ even when the underlying weights are the same.

export SERVED_MODEL_ID="COPY_THE_ID_FROM_MODELS"

curl -fsS "$OPENAI_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"$SERVED_MODEL_ID\",\"messages\":[{\"role\":\"user\",\"content\":\"What is 2 + 2? Answer briefly.\"}],\"max_tokens\":256}"

Expect a JSON chat completion with choices and token usage. Inspect choices[0].message for the answer and any separate reasoning output. A finish_reason of length means the output cap was reached. After a basic response works, validate the context lengths, tools, and concurrency your application needs.

Troubleshoot the deployment

A context-limit error on the iwant endpoint calls for shortening the prompt or output allowance to fit 8,192 tokens. The model’s larger advertised context does not override that server limit. When switching to Ollama or a GGUF server, also change the client’s model ID to the one returned by that endpoint.

If weights do not download, check repository access and token forwarding for the runtime you selected. If startup stalls, read the server logs and check storage and GPU memory before changing parallelism. For iwant provisioning or interrupted launches, follow the shared troubleshooting guide and check existing resources before launching another node.

Stop serving and remove cloud resources

Stopping a foreground server (Ctrl+C, ollama stop, or ending a container) does not delete a cloud VM. For iwant, remove the cluster with billing and teardown (iwant list, iwant down EXACT_CLUSTER_NAME). For vLLM, SGLang, or Docker you started on a rented host, stop the process and remove the host through your cloud workflow. The self-hosting hub summarizes idle-timer behavior and interrupted launches. Log token usage and GPU runtime when comparing self-hosted costs.

Last updated on
Self-hostingDeepSeek-V4-Flash