Measuring the Damage: Evaluations That Catch Quantization Errors

On all 1,319 GSM8K problems, eight quantized versions of Qwen3-8B scored within 0.7 points of the original; only the 2-bit one fell further. Read that way, 4-bit is free. It isn't: compared problem by problem, every format changed dozens of answers, and on a code benchmark most of them lost 4 to 8 points, FP8 included. This tutorial builds the four checks that showed it, with the code, so you can run them on your own model and formats.

The setup for everything below: an NVIDIA GB10, vLLM 0.31.0, Qwen3-8B in BF16 as the reference, and eight formats of the same model, the checkpoints of our size, speed, and quality sweep: FP8 (Qwen's block-scaled checkpoint), INT4 with AWQ, GPTQ, and plain round-to-nearest (group 128), NVFP4, and three GGUF files (Q4_K_M, Q3_K_M, Q2_K). Every evaluation is greedy, so differences come from the model, not from sampling.

Step 1: measure the noise first

Before blaming a format for a difference, know how much two runs of the same model differ. Greedy decoding on a serving engine is not bit-for-bit reproducible: requests are batched with whatever else is running, and a different batch shape changes the order of floating-point additions, which can flip a close choice between two tokens, and from there the rest of the answer.

So we served BF16 twice and ran everything twice:

Two lessons. Scoring a fixed text (log-probabilities over a prompt) is deterministic here, so any difference in it is the format's. Generating answers is not: 15 GSM8K answers change between identical runs. A format whose GSM8K results differ from BF16 by a handful of problems has shown nothing.

Step 2: token-level divergence, the cheap ranking

The most sensitive check doesn't generate anything. Send the same text to both models and compare, at every position, the probability each one gives to the possible next tokens. vLLM returns them with prompt_logprobs:

# scripts/gx10/quant_evals.py (abridged): the same 40 windows of 2,048 WikiText-2 tokens for every model
r = await client.completions.create(model="m", prompt=window_token_ids, max_tokens=1, temperature=0,
                                    extra_body={"prompt_logprobs": 20})
plp = r.model_extra.get("prompt_logprobs") or r.choices[0].model_extra.get("prompt_logprobs")
positions = [{tok: v["logprob"] for tok, v in pos.items()} for pos in plp[1:]]  # top 20 per position

For each position, two numbers against the reference's top 20:

• top-1 agreement: does the format rank the same token first?
• KL divergence, KL(BF16 ‖ format) = Σ p_BF16(t) · (log p_BF16(t) − log p_format(t)) over BF16's top 20 tokens. A token missing from the format's top 20 gets the format's 20th log-probability, an approximation that tends to underestimate the true KL.

(BF16's perplexity is 9.72, from the sweep.) KL and perplexity rank the formats almost the same way, but KL measures the right thing: how far the format's predictions moved from the reference, position by position, rather than how well either model predicts the text. It ran in a few minutes per format, with no answers to grade, with no answers to grade. Use it to sort candidates (on HumanEval, FP8 had the lowest KL and still lost 5.5 points, while GPTQ, at four times the KL, lost 0.6); it can't tell you whether a given shift matters for your task.

Step 3: compare answers in pairs, and test the difference

Accuracy on GSM8K was flat. The paired view, each problem compared with BF16's answer, tells a different story:

Every format changed 50 to 105 answers, three to seven times the noise floor, but in both directions, so the score stayed put. That is the pattern Dutta et al. called flips: the quantized model is a different model that happens to score the same. Whether the difference is real is a statistical question with a standard answer for paired results, McNemar's test: under "no difference", a problem that one model solves and the other doesn't is equally likely to go either way.

from math import comb

def mcnemar_p(lost: int, gained: int) -> float:
    """Exact two-sided McNemar test: probability of a split at least this uneven if both models were equal."""
    n, k = lost + gained, min(lost, gained)
    return min(1.0, 2 * sum(comb(n, i) for i in range(k + 1)) / 2 ** n)

print(mcnemar_p(77, 28))   # GGUF Q2_K on GSM8K: 1.853e-06
print(mcnemar_p(33, 25))   # NVFP4 on GSM8K: 0.36

On GSM8K, only Q2_K's loss is significant. Everything else is churn: you can't tell those formats from BF16 on this benchmark, which is not the same as their being as good.

Step 4: add a benchmark that punishes small errors

Math word problems tolerate a slightly different phrasing on the way to the answer. Code doesn't: one wrong token, a misnamed variable or a missing bracket, fails the tests. We ran HumanEval (164 Python functions, the original tests), each completion run in a separate Python process (python -I) in a temporary directory, with a 10-second timeout:

Here the losses are one-sided: formats lose problems BF16 solved and rarely gain any. FP8, the format usually called lossless, lost 10 and gained 1 (p = 0.012). GPTQ was the exception, statistically indistinguishable from BF16, while AWQ, at the same KL, lost 12. Taken one at a time, any of these p-values could be luck among eight formats tested; what is not luck is the direction: seven of eight formats lost more than they gained, and BF16 run twice lost none. The losses also cluster: ten problems were lost by four formats or more, so a handful of fragile tasks carries much of the difference. With only 164 problems, borderline cases (p between 0.05 and 0.1) need a larger code set before you conclude either way.

Step 5: watch the length of long answers

With thinking on, GSM8K answers run to about 1,100 tokens, and errors compound over them. Accuracy on 200 problems separated only Q2_K (88.5% against 96.0%, 16 lost, 1 gained, p = 0.0003). But the length of the reasoning moved, longer or shorter, for several formats:

A model that reasons 21% longer to reach the same answers (NVFP4) costs 21% more decode time and tokens per request, which can cancel part of the speed its smaller weights bought. Record output length next to accuracy for any reasoning workload.

The recipe

For a new format of a model you already serve:

1. Serve both the reference and the candidate with the same engine version, chat template, and generation settings.
2. KL on a fixed text (a few minutes): rank candidates; drop the obvious outliers.
3. Run the reference twice on your task suite: that is your noise floor.
4. Paired task evaluation on several hundred problems, including at least one unforgiving task close to your use: code, structured output, exact retrieval. Test lost-versus-gained with McNemar, not the accuracy difference.
5. For reasoning workloads, compare output lengths and truncations, not only answers.

Each format took 12 to 18 minutes for all four evaluations on our machine, and about an hour for the GGUF files, which run slowly on vLLM.

Gotchas

The chat template is part of the model. GGUF files and some community checkpoints ship their own templates; an evaluation that silently uses a different one measures the template. We passed the reference tokenizer to every GGUF run (--tokenizer Qwen/Qwen3-8B).

Truncation looks like a wrong answer. A reasoning answer cut at the token limit has no final answer to grade. Count truncations separately, and give long-reasoning evaluations a generous limit.

Running generated code needs isolation. HumanEval executes what the model wrote. Run it in a throwaway directory with a timeout, at least; in a container if the model or the prompts are untrusted.

One benchmark's noise is not another's. Our noise floor was 15 flips on GSM8K and 1 on HumanEval. Measure it for each task you use; it depends on the task's length and on how close the model's choices are.

Weights and activations are not the only thing a serving stack rounds: the KV cache is quantized too, and quantizing the KV cache applies the same checks to seven cache formats.