Speculative Decoding from First Principles

Generating one token takes one pass of the model, and that pass is limited by how fast the weights move from memory, not by arithmetic. So the GPU could check several tokens for nearly the price of one, if only it knew which tokens to check. Speculative decoding gets them from a cheap guesser and lets the big model verify all the guesses in a single pass. On Qwen3-8B, that made answers 2 to 3 times faster in our runs, without changing a single output token. Here is how, and where the gain disappears.

Spare compute in every step

The continuous batching piece measured it: one Qwen3-8B request on our test machine (an NVIDIA GB10) gets a token every 73 milliseconds, and that time is spent mostly reading 16 GB of weights. Processing 4 positions instead of 1 in the same pass reads the same weights, so it costs almost the same time. The question is what to put in those extra positions.

Draft, verify, keep what matches

Speculative decoding fills them with guesses:

1. Draft. A cheap drafter proposes the next few tokens, say 3.
2. Verify. The big model reads the context plus the 3 drafts in one pass. Because it is a transformer, that one pass gives its own choice at every position: after the context, after the first draft, after the second, and after the third.
3. Keep. Compare, left to right. Drafts are kept as long as they match the big model's choice; at the first disagreement, the big model's own token replaces the draft, and everything after it is discarded. If all 3 match, the big model's choice after the last draft comes free as a fourth token.

Every step produces at least one token, the big model's own, so speculation can never produce fewer tokens per pass than plain decoding. It can still be slower: the drafter costs time, and a verification pass over 4 positions costs more than a pass over 1 once the batch is large.

The output does not change. Under greedy sampling, a draft is kept only if it equals the big model's top choice (vLLM's check), so the text is the same token for token. Under random sampling, the method of Leviathan et al. and Chen et al. keeps a draft with probability min(1, p/q), where p is the big model's probability for that token and q the drafter's (L884), and on rejection samples from what is left of the big model's distribution, max(0, p − q) (L1061). The result follows the big model's distribution exactly. Speculation is an optimization, not an approximation.

How many tokens per step, and what it costs

If each draft is accepted with probability α, independently, and the drafter proposes k tokens, a step yields on average (1 − α^(k+1)) / (1 − α) tokens. At α = 0.7 and k = 3, that is 2.5 tokens per pass of the big model. Whether that becomes a 2.5 times speed-up depends on what the step costs:

Three things to try:

1. Lower the acceptance rate to 30%. Most steps produce one or two tokens; with a small draft model, the drafter's cost eats most of the gain.
2. Propose 8 tokens instead of 3 at 70% acceptance. Expected tokens per step barely rise (2.5 to 3.2): long drafts rarely survive to the end, which is why engines propose 3 to 5.
3. Move to 128 requests. The verification pass now carries 512 tokens and becomes compute-bound: it costs more than a normal step, and the speed-up falls below what the acceptance rate promises. Speculation helps most when the GPU is underused, which is exactly when there are few requests.

The widget is a model calibrated on our machine: a step costs at least the 72 ms it takes to read the weights, and 0.235 ms per token once the batch is large enough to be compute-bound (the same numbers as in the chunked prefill piece).

Four kinds of drafter

The drafter is the whole game: it must be cheap and agree with the big model often.

• n-gram lookup. No model at all: search the prompt for the last few tokens and propose what followed them there. Free, and very good when the answer copies the input (code edits, document rewriting, RAG answers that quote their sources), useless otherwise.
• A small draft model from the same family, sharing the tokenizer: here Qwen3-0.6B for Qwen3-8B. Good agreement, but it is a real model with its own forward passes and its own KV cache.
• EAGLE-style heads. A small network trained to predict the big model's next tokens from the big model's own hidden states (EAGLE-3), so it starts from what the big model already computed. Cheaper than a draft model, but it must be trained for each target model.
• MTP heads built into the model. DeepSeek V3, GLM, Qwen3.5-class and other recent models ship multi-token-prediction layers trained with the model; engines use them as built-in drafters.

In vLLM 0.31.0 they are all --speculative-config methods: ngram, draft_model, eagle/eagle3, model-specific MTP types, and newer ones (suffix, dflash, dspark) (config). SGLang 0.5.21 offers the same family through --speculative-algorithm (NGRAM, STANDALONE for a draft model, EAGLE, EAGLE3, NEXTN for MTP heads, and more).

Measured: Qwen3-8B, two workloads, four methods

We ran vLLM 0.31.0 with Qwen3-8B (BF16) on the GB10, 2026-10-06, with greedy sampling, thinking turned off, and 3 proposed tokens. Two real workloads from vLLM's benchmark tool: MT-Bench (open-ended chat questions) and InstructCoder (edit this code), each at 1, 8, and 32 requests in flight. For example, the EAGLE-3 run:

$ vllm serve Qwen/Qwen3-8B --gpu-memory-utilization 0.85 --speculative-config \
    '{"method": "eagle3", "model": "RedHatAI/Qwen3-8B-speculator.eagle3", "num_speculative_tokens": 3}'
$ vllm bench serve --backend openai-chat --endpoint /v1/chat/completions --model Qwen/Qwen3-8B \
    --dataset-name hf --dataset-path likaixin/InstructCoder --max-concurrency 1 --num-prompts 10 \
    --temperature 0 --extra-body '{"chat_template_kwargs": {"enable_thinking": false}}'

Acceptance comes from vLLM's own counters (vllm:spec_decode_num_accepted_tokens, ..._num_draft_tokens, ..._num_drafts); tokens per step include the big model's own token. Every request completed in every run.

What the numbers say:

1. The workload decides first. Code edits copy much of their input, so every method does well; open chat is harder to guess. The same EAGLE-3 head is accepted 75% of the time on code edits and 44% on chat.
2. n-gram is free money on copy-heavy work, and nothing else. 2.3 times faster on code edits, 1.2 times on chat. It costs nothing, so turning it on for an editing or RAG service is an easy win.
3. A good drafter beats a cheap one, until it doesn't. The 0.6B draft model is accepted more often than EAGLE-3 (it is a full model of the same family) but costs more per step, so the two end up within a few percent of each other: a tie on chat with one request, EAGLE-3 slightly ahead everywhere else.
4. The gain shrinks as the batch grows. At 32 requests, every method gives up 13 to 26% of its single-request speed-up: verification passes get heavier and the GPU has less spare capacity. On a bigger GPU or with more requests, it shrinks further.
5. Acceptance falls along the draft. EAGLE-3's first drafted token was accepted 84 to 90% of the time on code edits, its third 60 to 67%: each guess builds on the previous ones.

When to turn it on

• Interactive, latency-sensitive serving with moderate load (a coding assistant, an agent loop, a chat with few concurrent users): yes. Pick an EAGLE-3 or MTP drafter if one exists for your model, a same-family draft model otherwise, and n-gram if your answers quote their inputs.
• Throughput-bound batch serving at high concurrency: measure first. The GPU is already busy, and speculation can cost more than it saves.
• Tune num_speculative_tokens from your own acceptance counters, not from defaults: 3 is a good start; go higher only if late positions are still accepted often.
• Check what your drafter was trained on. The EAGLE-3 head we used was trained with thinking turned off; reasoning traces or another language can lower acceptance.

Next

Speculative decoding saves passes over the weights. NVFP4 makes each pass read fewer bytes in the first place: four-bit weights on Blackwell GPUs, the next piece.