Qwen3.8-Flash-Next is Alibaba's preview of the Qwen4 serving stack. Four aspects of the model shape how you deploy it:
- It is a 125B-parameter ultra-sparse mixture-of-experts model with only 6B parameters active per token.
- Its native context is 262,144 tokens, extensible to 1M with YaRN.
- It ships with multi-token prediction (MTP) and a vision encoder.
- It pairs Gated DeltaNet (GDN) linear-attention layers with Qwen Sparse Attention (QSA), which is what makes very long context cheap to serve.
Everything below was verified end-to-end on Runpod using 4× H200, via the OpenAI Python SDK.
Choosing the deployment
Runpod Hub's vLLM worker template (runpod-workers/worker-vllm, currently v2.26.0) ships vLLM 0.28.0, and Qwen3.8-Flash-Next requires vLLM 0.29 or newer.
The architecture (GDN, QSA, and n-gram embeddings) exists only in that build and later, so the Hub's one-click path does not serve this model today. vLLM publishes a dedicated image for it, vllm/vllm-openai:qwen38-flash-next, and we validated that image live on 2026-09-03 in two configurations.
You can run it as a load-balancing Serverless endpoint or as a GPU pod. The load-balancing endpoint is direct HTTPS to vLLM's OpenAI server, with streaming, autoscaling, per-second billing and no queue, which suits API-style traffic. A pod runs the same image and flags behind Runpod's HTTP proxy, and it suits interactive work and debugging because there is no health watchdog, so a slow first boot is not fatal.
Hardware
We used the official FP8 checkpoint Qwen/Qwen3.8-Flash-Next-FP8, roughly 173 GiB of weights across 131 shards, on NVIDIA H200s with 141 GB of HBM3e each. The official vLLM recipe supports TEP8, tensor-plus-expert parallelization across 8 GPUs. Eight-wide H200 nodes were capacity-constrained during our test, so we ran the other supported configuration, TP4 with expert parallelism on 4× H200. That gives 564 GB of HBM, which holds the weights, the KV cache, the vision tower and the MTP module with a large KV budget left over: vLLM reported 37.28 GiB of KV cache per GPU, about 3.02M tokens, enough for 11.5 concurrent sequences at the full 262K context.
On Hopper you must pass --enable-expert-parallel --moe-backend triton; plain tensor parallelism is incompatible with this checkpoint's 128-wide quantization blocks.
Cost (measured config)
| Path | GPU | Billed as | $/hr while running |
|---|
| Pod (Secure Cloud) | 4× H200 | $5.29/GPU/hr | $21.16/hr |
| Serverless worker | 4× H200 | $5.93/GPU/hr | $23.72/hr |
Serverless bills only while a worker is running, so bursty API traffic is usually cheaper there. Steady or interactive workloads favor a pod.
Creating the load-balancing endpoint
Select Serverless, then New Endpoint, then choose a custom image. These are the settings we validated:
| Setting | Value | Why |
|---|
| Endpoint type | Load balancing | Direct HTTP to vLLM with no queue handler in front |
| Image | vllm/vllm-openai:qwen38-flash-next | The only build with the new architecture (vLLM 0.1.dev20073) |
| GPU pool | H200 (HOPPER_141) × 4 | See the weights and KV math above |
| Min host CUDA | 13.0 | The image requires CUDA ≥ 13.0 hosts (NVIDIA_REQUIRE_CUDA) |
| Container disk | 400 GB | 173 GiB of weights plus the image and HF cache |
| Workers | min 0 / max 1 (min 1 while testing) | Scale to zero in production; force a boot while validating |
| Idle timeout | 3600 | Keeps the worker alive mid-test |
Container start args (the image entrypoint is vllm serve):
Qwen/Qwen3.8-Flash-Next-FP8 \
--tensor-parallel-size 4 --enable-expert-parallel --moe-backend triton \
--gpu-memory-utilization 0.85 --max-num-seqs 256 --enable-prefix-caching \
--no-enable-flashinfer-autotune \
--enable-auto-tool-choice --tool-call-parser qwen3_xml \
--reasoning-parser qwen3 \
--enforce-eager
--enforce-eager skips torch.compile and CUDA-graph capture. Our pod boot spent 46.6 s compiling and 36.2 s capturing 86 CUDA graphs; eager moves that time off the boot path so the endpoint passes the platform health window. Add --gdn-prefill-backend triton to also skip the FlashInfer GDN JIT that otherwise hits the first request (see the performance section).
Environment variables:
| Env | Value | Why |
|---|
NCCL_NVLS_ENABLE | 0 | Required. NVLS/fabric init fails on these nodes; without it all 4 TP ranks die at ncclCommInitRank before any weight download |
PORT / PORT_HEALTH | 8000 | LB endpoints default to port 80; vLLM listens on 8000 |
HEALTH_CHECK_PATH | /health | The platform default polls /ping; vllm-openai dev images answer both, but pinning /health is image-agnostic |
HF_HUB_ENABLE_HF_TRANSFER | 1 | Fast weight download (deprecated alias; HF_XET_HIGH_PERFORMANCE=1 is the modern name) |
HF_TOKEN | yours | Avoids HF rate-limit stalls on big downloads |
The Runpod gateway authenticates requests with your Runpod API key, so do not add --api-key to vLLM on this path. The gateway cannot know the value, and every request would 401 inside the worker. On a pod there is no gateway, so you can set --api-key there.
Measured on September 3, 2026, on a four-GPU H200 worker, startup took about 11 minutes from scale-up to the first successful health check: roughly 4.5 minutes to pull and extract the image, under a minute for NCCL initialization (with NCCL_NVLS_ENABLE=0), 167.7 seconds to download roughly 173 GiB of model weights to a warm host, 38 seconds to load the shards, 35.3 seconds to initialize the engine in eager mode and 19.2 seconds for multimodal warmup.
On a fully cold host, allow 20–25 minutes for the Hugging Face download. Monitor progress in the Workers tab or with stream-worker-logs; the worker is ready when the logs report that application startup is complete.
Making a request
Once the worker is healthy, call the endpoint with the standard OpenAI Python client. Set RUNPOD_API_KEY in your environment, replace <ENDPOINT_ID> with your Runpod endpoint ID and make sure the model value matches the model your deployment serves.
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["RUNPOD_API_KEY"],
base_url="https://.api.runpod.ai/v1",
timeout=600.0,
)
response = client.chat.completions.create(
model="Qwen/Qwen3.8-Flash-Next-FP8",
messages=[{"role": "user",
"content": "Explain how a mixture-of-experts model works in a couple of sentences."}],
max_tokens=2048,
temperature=1.0, top_p=0.95, # model's recommended defaults
extra_body={"top_k": 20},
)
print(response.choices[0].message.content)
print(response.usage)
Response:
A mixture-of-experts model combines multiple specialized submodels, or
"experts," along with a gating mechanism that decides which experts to use
for each input. This allows the model to allocate different parts of the
problem space to the experts best suited for each example.
usage: prompt=66 completion=101 (reasoning_tokens=45), total=167
Thinking is on by default. usage.completion_tokens_details.reasoning_tokens counts the reasoning trace, and with --reasoning-parser qwen3 the visible content excludes it. Streaming works with the standard stream=True.
Non-thinking mode, tool calling and vision
All six checks in scripts/test_endpoint.py passed against the exact configuration above on 2026-09-03.
Instruct mode works by passing extra_body={"chat_template_kwargs": {"enable_thinking": false}} with the model's recommended sampling for that mode (temperature=0.7, top_p=0.8, presence_penalty=1.5, top_k=20). It answered a mini-math question with just 391 and no reasoning preamble.
Tool calling works through the qwen3_xml parser. With a get_weather tool defined, response.choices[0].message.tool_calls[0] came back as get_weather({"city": "Reykjavik"}).
Vision needs no extra flags because the tower loads by default. We sent a data-URI PNG of a red, white and blue striped field, and the model described it as a tricolor flag with three equal vertical stripes of red, white and blue from left to right.
Measured performance
| Metric | Pod (compiled) | LB endpoint (eager) |
|---|
| Cold start (create/scale-up → serving) | 7 min 38 s | ~11 min (incl. ~4.5 min image pull; 6 min 26 s from container start) |
| NCCL init (4 ranks) | 0.4–0.6 s | under 1 min (same NCCL_NVLS_ENABLE=0 fix) |
| Weights download (172.78 GiB, 131 shards) | ≤ 4.5 min (warm host) | 167.7 s (warm host) |
| Engine init incl. torch.compile | 110.5 s (46.6 s compile) | 35.3 s (eager: no compile or graphs) |
| CUDA graph capture (86 graphs) | 36.2 s | skipped |
| Multimodal warmup | 18.9 s | 19.2 s |
| KV budget per worker | 3.02M tokens → 11.5× at 262K | 5.73M tokens → 21.8× at 262K (no CUDA graphs, so more memory for KV) |
| TTFT (streaming) | 3.60 s first request (FlashInfer GDN JIT), 3.12 s warm | 18.1 s first request (eager Triton kernel JIT: fused_moe/QSA), not yet re-benchmarked warm |
| Throughput (single stream) | 58 tok/s (answer incl. 424 reasoning tok), 137.5 tok/s decode | ~8–10 tok/s on first requests while kernels JIT |
| Vision call | 2.6 s | 128.4 s on the retest (eager; first-ever image encode) |
Eager mode gets the server reachable 5–7 minutes sooner and serves its early requests an order of magnitude slower than the compiled engine, because every Triton kernel JITs cold on first use; even at steady state, eager decode runs well below the compiled pod's 137 tok/s. You have three ways out: accept eager for the first worker and let REQUEST_COUNT scaling bring the others up the same way, run compiled and persist VLLM_CACHE_ROOT on a network volume so compile artifacts carry across workers, or use pods for anything latency-sensitive.
Cold-start levers, in order of effort:
--enforce-eager removes compile and graph capture from the boot path.- Keep
workersMin: 1 during business hours so the second boot on a host reuses the HF cache. Our warm-host download was 167.7 s for 173 GiB; budget up to 20–25 minutes on a cold host. - Attach a network volume with
VLLM_CACHE_ROOT (and HF_HOME) so the compile cache and weights survive worker recycling. The full playbook is in Cut vLLM Cold Starts on Runpod Serverless. - On the request side,
--gdn-prefill-backend triton avoids the FlashInfer GDN JIT that made our first-request TTFT 3.60 s against 3.12 s warm.
Disable NVLS before your first boot
Every H200 node we landed on failed NVLS (NVLink SHARP fabric) initialization: all four NCCL ranks died at ncclCommInitRank with unhandled cuda error before a single byte of weights downloaded. The fix is a single environment variable: NCCL_NVLS_ENABLE=0. With it set, all ranks initialize in about 0.5 s via P2P/CUMEM and serving proceeds. You will also see SymmMemCommunicator: symmetric memory multicast operations are not supported warnings; those are the expected consequence of disabling NVLS and are safe to ignore. Save yourself four crash-looped pods and set the variable on day one.
Conclusion
Until the Hub worker ships vLLM 0.29 or newer, both paths run the same vllm/vllm-openai:qwen38-flash-next image with the same flags: the load-balancing Serverless endpoint for API traffic and a pod for interactive work and maximum single-stream throughput. One environment variable, NCCL_NVLS_ENABLE=0, made the difference between a crash loop and a clean boot. With the configuration tables above, you can go from zero to an OpenAI-compatible API serving 262K-context Qwen3.8-Flash-Next in about eleven minutes.
Next steps
Runpod Serverless runs your container as an autoscaling endpoint that scales to zero and bills by the second.
Resources