Qwen3.8-Flash-Next, Layer by Layer

Qwen3.8-Flash-Next activates 6 billion parameters per token, a third of the 17 billion of its predecessor, the 397-billion-parameter Qwen3.7-Plus, was trained on a third of the tokens with about a ninth of the compute, and still leads that predecessor on 8 of 14 pre-training benchmarks. No single trick explains it. Released in August 2026 as a preview of Qwen's next architecture, the model combines five decisions, and the Qwen team's paper measures each one.

The model at a glance

Values from the config and model card. Engines know it as Qwen4ExpForConditionalGeneration.

Three quarters of the layers keep a fixed-size state; only 12 keep a cache that grows, and those read a selected 2,048 tokens.

Token mixing: a fixed state and a selective cache

Gated DeltaNet (GDN), in 36 of the 48 layers, is linear attention: instead of keeping every past key and value, each layer folds the past into a fixed-size matrix per head and updates it with every token. "Delta" is the update rule: the layer reads what the state currently predicts for the new key, and writes only the difference from the true value; "gated" adds a learned decay that lets the state forget. The cost per token doesn't grow with the context, and neither does the memory. Our hybrid attention piece sizes these states in detail.

The other 12 layers keep real keys and values, so the model can still retrieve an exact token from far back. In the released model they use Qwen Sparse Attention (QSA): each query reads only 2,048 of the past tokens, chosen by a small indexer.

The indexer ranks blocks of four tokens, so its own cost is a quarter of scoring every token, and attention reads 2,048 tokens however long the context.

The indexer is a miniature attention: 4 query heads and 1 key head of 128 dimensions. It averages the keys of every 4 consecutive tokens into one, scores those blocks for the current query, and keeps the best 512 blocks, 2,048 tokens. DeepSeek's V3.2 indexer scored every token; scoring blocks of four makes this indexer four times cheaper than scoring every token would be.

QSA was added late. The model was pre-trained with ordinary full attention in those 12 layers; continued pre-training then trained the indexer alone for about 2 billion tokens, to imitate where full attention was looking, and the whole model with QSA for about 200 billion more, at 256K-token sequences. The result was better, not just cheaper: the long-context RULER score above 512K tokens rose from 90.1 to 93.0, multi-round coreference (MRCR) at 1M tokens from 20.7 to 26.4, and short-context benchmarks moved up slightly (75.9 to 76.8 on average). At 1M tokens, Qwen's own attention kernels run 7.6 times faster in prefill and 4.9 times faster in decode than full attention, for the attention module alone (the paper's kernel benchmark).

One decision went the other way: the full-attention layers keep rotary position embeddings. A variant without them matched during pre-training but produced more never-ending generations after post-training.

The gated residual: four streams, one gate

A transformer normally has one residual stream: each layer reads it, through a normalization, and adds its output back. Qwen3.8-Flash-Next widens it to four branches. Each layer reads a mix of the four through an elementwise sigmoid gate predicted from all of them, and writes its output back to each branch with a per-branch scalar predicted for each token.

Qwen keeps the four streams of hyper-connections but drops the matrix that mixes them, which saves a full read of the residual state per block.

This is close to DeepSeek's hyper-connections (V4 and V4.1 also use four streams), with one difference that matters for speed: Qwen drops the matrix that mixes the streams before every layer. In the paper's 25B-parameter test (3B active, 560 billion tokens), the gated residual lowered the loss from 1.617 to 1.590 and raised the benchmark average from 50.9 to 54.7. In a 28-layer test it also matched "attention residuals", which let each layer read from all earlier layers.

The gate's bigger contribution is stability. In a stress test at three times the optimal learning rate, adding the same gate (GatedNorm) to an ordinary single-stream model cut loss spikes from 32.0 to 3.2 per 10,000 steps; at four times, the run with the full gated residual had none. For serving, the four branches mean four times the residual bytes read and written per layer; Qwen stores them in FP8, which halves those bytes with almost no loss in quality.

Experts and memory

The mixture-of-experts layers are fine-grained: 512 experts per layer, each with an intermediate size of only 640, and every token uses 10 of them plus one shared expert. Many small experts give the router more combinations to choose from at the same cost per token; the price is more routing and more, smaller matrix multiplications, which is where expert parallelism and grouped GEMM kernels earn their keep.

The fifth component adds capacity that costs almost nothing per token: an n-gram embedding table at layer 2. It is the same mechanism as DeepSeek's Engram: the last 2 and 3 tokens are hashed by 16 hash heads into tables of about 20 million rows each, the 16 retrieved rows of 160 values form a 2,560-value vector, and a gate compares it with the hidden state before it enters the four residual branches. 16 × 20 million × 160 gives the 51 billion parameters, more than 40% of the backbone's size. Qwen dropped one Engram step: it found that normalizing tokens (merging "Apple" and "apple") gave no consistent gain.

The table's size came out of a sweep: growing it from 20 to 200 times the 250K-token vocabulary kept lowering the loss, but downstream scores flattened; the shipped table sits at about 80 times.

Training: a ninth of the compute

The paper's comparison is with Qwen3.7-Plus, 397B parameters with 17B active (the abstract calls it "the 397B-A17B predecessor"): Flash-Next has about a third of the active parameters, was trained on about a third of the tokens, with roughly a ninth of the training FLOPs, leads on 8 of 14 pre-training benchmarks and trails by at most 2.6 points on the rest. It beats the dense Qwen3.8-27B base model on all 14.

The ninth is arithmetic: a third of the active parameters times a third of the tokens. The result to explain is that this smaller budget matched the bigger model, and two things did it. The architecture spends compute where it pays: 6 billion active parameters, linear layers for most of the mixing, a free lookup for local patterns. And the training recipe avoids the instabilities that usually force smaller learning rates or restarts: the Muon optimizer for weight matrices, no batch-size warm-up (it cost 18.8% more steps for nothing), learning rate and batch size from a refitted scaling law, and the gated residual. The production run had no loss spikes and needed no qk-clip or SwiGLU-clip. Over its first 276 billion tokens, the gated residual and the rest of the recipe lowered the loss by 0.058 against a run with the previous (Qwen3.5) architecture and Muon alone, 0.026 of it from the gated residual.

The paper's measurements behind each choice, with their setups:

Serving it

The checkpoint is 335 GiB in BF16 and 173 GiB in FP8, which counts the backbone, the n-gram table and the MTP layer. The FP8 checkpoint fits on a single 288 GB GB300; the vLLM recipe recommends four of them with tensor and expert parallelism for serving.

Per request, the model holds three kinds of state:

• GDN states: 36 layers × 48 heads × 128 × 128 values, kept in FP32 as the config specifies (mamba_ssm_dtype): 108 MiB per request, whatever the context length (54 MiB if you force BF16 with --mamba-ssm-cache-dtype bfloat16).
• QSA keys and values: 12 layers × 2 KV heads × 256 dimensions × 2 (K and V). With an FP8 cache that is 12 KiB per token, 3 GiB for a full 262,144-token request. The same model with full attention in all 48 layers would need four times that.
• Indexer keys: one compressed 128-value key per 4 tokens per QSA layer, a few hundred bytes per token.

Per request, the fixed state is small and the cache is light; the table is the one big object, and it can live in host memory.

Both engines can keep the n-gram table in host memory: vLLM v0.32.0 by default, SGLang by default for the BF16 checkpoint. vLLM serves it through its Engram code path, where Qwen's version is "the same mechanism, called PLE", and keeps it in pinned host memory unless you pass --engram-config.cpu_offload false. SGLang keeps each GPU's shard of the table in pinned host memory and copies the 16 rows each token needs while the first layer runs. On 4 H200s, the SGLang team measured at launch (August 2026) the offload taking per-GPU weights from 83.9 to 60.5 GiB and KV capacity from 1.84 to 3.28 million tokens, with throughput unchanged (BF16, MTP on).

The MTP layer drafts tokens for speculative decoding. The vLLM recipe drafts 3 (--speculative-config '{"method":"mtp","num_speculative_tokens":3}'). SGLang reuses one QSA block selection across all the draft steps of an iteration, and at launch reported 540 tokens per second for a single user (batch size 1) on 4 B200s with the NVFP4 checkpoint, at an accept length of 3.3 including the bonus token, with the SGLang team's own NVFP4 checkpoint. Both engines have tuned it since: SGLang v0.5.22 decodes it 12% faster on B200, and vLLM v0.32.0 carries its MTP acceptance across prefill/decode disaggregation.

A launch on 4 GB300s, from the vLLM recipe:

# vLLM recipe (vLLM >= 0.30.0, verified on nightly): Qwen3.8-Flash-Next FP8 on 4x GB300, TEP4
vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --attention-config.indexer_kv_dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

--enable-expert-parallel splits the 512 experts across the GPUs instead of slicing each one; --attention-config.indexer_kv_dtype fp8 stores the indexer's keys in FP8 too. We haven't run this command; it is the recipe's.

What it says, and what it changes for you

The direction is clear, and it is shared: DeepSeek's V4.1 made most of the same bets in the same year (sparse attention with an indexer, four residual streams, many routed experts, an n-gram memory in host RAM, a trained draft head). The open frontier is converging on models where most layers keep a small fixed state, a few layers read a selected slice of a long context, and capacity moves to parameters that are looked up rather than computed.

For the people who serve them, that changes the arithmetic:

• Memory per request is light, and fixed for short requests. 108 MiB of GDN state plus 12 KiB per token of QSA cache: the cache only overtakes the state past about 9,200 tokens, and a full 262K-token request costs 3 GiB instead of the 12 GiB all-full-attention would take. Below about 9,000 tokens, the fixed-state pool (256 slots × 108 MiB, about 27 GiB with the recipe's --max-num-seqs 256) caps concurrency before context length does.
• Host memory and the CPU-GPU link are part of the deployment. Plan about 51 GB of host RAM per model instance for vLLM's FP8 table, about 95 GiB for SGLang's BF16 table.
• Engine releases matter. Each component needs its own kernels and scheduler support: SGLang v0.5.22's release notes list a 12% faster decode for this model on B200, so test on the latest vLLM or SGLang before tuning flags.

Gotchas

Plain TP8 fails with the FP8 checkpoint. Each expert is 640 wide and quantized in 128-wide blocks; split 8 ways, a GPU would get 80, less than one block. The recipe uses expert parallelism instead (TEP8) on 8 H200s or H100s, which keeps each expert whole.

Keep --max-num-seqs at 256 in vLLM. The recipe sets it to avoid a capacity error for the GDN state pool ("Mamba cache") at startup: fixed states are allocated per sequence slot up front.

No pipeline parallelism with the n-gram table. vLLM's recipe lists it as unsupported; use tensor and expert parallelism within a node.

The offload is NVIDIA-only for now in vLLM. On AMD, the table sits in GPU memory, so budget 48 GiB (FP8) of HBM for it.

Benchmark MTP on real text. With random tokens, drafts are never accepted and MTP looks like pure overhead; the recipe benchmarks with --dataset-name speed_bench.

The full 262K context in one request is untested. The recipe validated startup and bounded evaluations, not a single 262,144-token request; test your longest prompts before promising them.

Next: speculative decoding, to see what the MTP layer buys, and the KV cache, sized, for the arithmetic behind the 12 KiB per token.