Self-host Ling-3.0-flash-fp8: GPU requirements and deployment

Self-host the FP8 inclusionAI/Ling-3.0-flash-fp8 checkpoint using iwant, vLLM, or SGLang. iwant launches a full eight-H200 cloud node. The direct-runtime recipes below use smaller H200 layouts, so choose a path based on the hardware you have and the deployment work you want to manage.

GPU and storage requirements

iwant requests eight H200s, at least 16 vCPUs, at least 64 GB of host RAM, and a 150 GB boot disk, with weights staged on attached local storage. The official vLLM FP8 configuration uses two H200s; the SGLang FP8 configuration below uses four H200s with expert parallelism. The model’s 262,144-token context capability is separate from the effective serving limit and available KV cache on a particular launch.

For the raw-runtime layouts and checkpoint variants, consult the linked vLLM and SGLang recipes below. For iwant, preview the selected cloud resources with iwant up ling-3.0-flash-fp8 --infra gcp --dry-run --yes before spending money.

Deploy with iwant

Complete iwant installation and cloud authentication once. Your GCP project needs billing enabled, GPU quota, and available capacity. Then launch by model name:

iwant up ling-3.0-flash-fp8

The CLI guides you through the launch choices and selects the serving configuration automatically. Choose a real launch rather than the dry-run preview when prompted. For a scripted on-demand launch, use iwant up ling-3.0-flash-fp8 --infra gcp --yes instead. You do not need to choose an image or assemble engine flags.

Save the printed Server URL and API key after startup. Readiness means the models API responded; use the first request below to check inference. GCP is currently supported. For download authentication or provisioning problems, use the shared setup and troubleshooting guide.

Serve with vLLM

Use two H200s on a compatible host and complete vLLM installation. The official Ling recipe  specifies vLLM 0.28.0 or newer and provides this serialized-FP8 TP2 path. Its TP4 plus expert-parallel configuration is another documented choice; neither requires changing the checkpoint name.

Replace YOUR_SERVER_KEY with your chosen server key before starting the process. Run this on the GPU host, with any required HF_TOKEN set in that shell.

export MODEL_API_KEY="YOUR_SERVER_KEY"
 
vllm serve inclusionAI/Ling-3.0-flash-fp8 \
  --trust-remote-code --dtype bfloat16 \
  --tensor-parallel-size 2 --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill --enable-prefix-caching \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-auto-tool-choice --tool-call-parser ling3 \
  --reasoning-parser ling3 \
  --host 127.0.0.1 --port 8000 --api-key "$MODEL_API_KEY"

The example binds to loopback. Test from another terminal on the same host, or use an SSH tunnel to reach its port. Stop the foreground process with Ctrl+C when finished; removing a rented host is a separate cloud action.

Serve with SGLang

The official SGLang Ling cookbook  provides an FP8, four-H200, high-throughput configuration using TP4 and expert parallelism. Use its dedicated image on a Docker/NVIDIA Container Toolkit host. This selected path leaves speculative decoding off; the cookbook also covers NEXTN and DSpark configurations.

Use your chosen server key and set HF_TOKEN if the download requires authentication. This is an alternative server: stop any earlier process using port 8000 before starting it.

export MODEL_API_KEY="YOUR_SERVER_KEY"
mkdir -p "$PWD/hf-cache"
 
docker run --rm --gpus all --ipc=host --shm-size 32g \
  -p 127.0.0.1:8000:8000 \
  -v "$PWD/hf-cache:/root/.cache/huggingface" \
  -e HF_TOKEN \
  lmsysorg/sglang:dev-Ling-3.0-flash \
  sglang serve \
  --model-path inclusionAI/Ling-3.0-flash-fp8 \
  --tp 4 \
  --ep-size 4 \
  --mem-fraction-static 0.8 \
  --host 0.0.0.0 --port 8000 \
  --api-key "$MODEL_API_KEY"

Wait for startup, then run the shared API check from the same host or through an SSH tunnel. Ctrl+C stops this foreground server; it does not remove the underlying VM.

Verify the model API

After any launch path above, confirm the server with OpenAI-compatible HTTP. Set OPENAI_BASE_URL to the base URL including /v1 (iwant prints this; vLLM and SGLang examples use http://127.0.0.1:8000/v1; Ollama uses port 11434; LocalAI uses 8080). Set OPENAI_API_KEY to the key from iwant or the value you chose when starting the server. Run curls on the GPU host or through SSH when the process binds to loopback.

export OPENAI_BASE_URL="YOUR_SERVER_URL_WITH_V1"
export OPENAI_API_KEY="YOUR_SERVER_API_KEY"

curl -fsS "$OPENAI_BASE_URL/models" \
  -H "Authorization: Bearer $OPENAI_API_KEY"

Copy the served id from the response into SERVED_MODEL_ID.

iwant typically returns inclusionAI/Ling-3.0-flash-fp8. Model aliases can differ even when the underlying weights are the same.

export SERVED_MODEL_ID="COPY_THE_ID_FROM_MODELS"

curl -fsS "$OPENAI_BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"$SERVED_MODEL_ID\",\"messages\":[{\"role\":\"user\",\"content\":\"What is 2 + 2? Answer briefly.\"}],\"max_tokens\":256}"

Expect a JSON chat completion with choices and token usage. Inspect choices[0].message for the answer and any separate reasoning output. A finish_reason of length means the output cap was reached. After a basic response works, validate the context lengths, tools, and concurrency your application needs.

Troubleshoot the deployment

Use the -fp8 checkpoint ID when calling this deployment. BF16, FP4, INT4, and vision-language Ling variants require their own launch settings. For reasoning or tool parsing issues, compare the installed engine and startup flags with its recipe; the vLLM path here explicitly uses ling3 parsers.

If weights do not download, check repository access and token forwarding for the runtime you selected. If startup stalls, read the server logs and check storage and GPU memory before changing parallelism. For iwant provisioning or interrupted launches, follow the shared troubleshooting guide and check existing resources before launching another node.

Stop serving and remove cloud resources

Stopping a foreground server (Ctrl+C, ollama stop, or ending a container) does not delete a cloud VM. For iwant, remove the cluster with billing and teardown (iwant list, iwant down EXACT_CLUSTER_NAME). For vLLM, SGLang, or Docker you started on a rented host, stop the process and remove the host through your cloud workflow. The self-hosting hub summarizes idle-timer behavior and interrupted launches. Log token usage and GPU runtime when comparing self-hosted costs.