How Much Does Each Bit Cost? Size, Speed, and Quality, Measured

Qwen3-8B can be served from the same Hugging Face page in more than a dozen forms: the original BF16, two FP8 checkpoints, AWQ, GPTQ, NVFP4, and a list of GGUF files from 8 bits down to 2. Each promises a smaller, faster model at some cost in quality. I loaded seventeen of them, one after the other, on the same GPU with the same engine, and measured what each one actually costs and buys.

Setup

• Hardware and engine: one NVIDIA GB10 (Blackwell), vLLM 0.31.0, the same launch settings for every format.
• Size: the weight memory vLLM reports at startup, and the KV cache room it has left.
• Speed: vllm bench serve with 512-token prompts and 256-token answers, one request at a time and 32 at once; and the time to the first token of an 8,000-token prompt.
• Quality: all 1,319 GSM8K math problems, answered greedily with thinking off; and WikiText-2 perplexity, computed on exactly the same 298,862 tokens for every format.
• Checkpoints: public ones where a trusted publisher exists (Qwen, Red Hat AI, Unsloth for GGUF); otherwise made with llm-compressor (INT4 round-to-nearest, INT8 W8A8, MXFP4); MXFP8 and NF4 quantized by vLLM at load time.

The results

All runs completed with zero failed requests. The GGUF rows use vLLM's GGUF plugin, which is not what most GGUF users run (llama.cpp); their speeds say more about the plugin than about the format, and this site does not benchmark llama.cpp.

Size and decoding speed: the bytes decide

Decoding one request reads every weight once per token, so its speed is set by how many bytes the weights take (The Roofline for LLMs). The measurements follow that rule closely: every format sits near the curve "211 GiB/s divided by the weight size", from BF16 at 13.8 tokens per second to 4-bit at 41. vLLM's 8-bit formats are 1.6 to 1.8 times faster than BF16, its 4-bit formats 2.7 to 3 times.

None of the 4-bit models is a quarter of BF16: 5.7 to 6.0 GiB, not 3.8. The embeddings and the output layer stay in 16 bits in these checkpoints, and for Qwen3-8B they alone are about 2.3 GiB (Quantization, Mapped).

The memory saved goes to the KV cache: from 626,416 cached tokens with BF16 weights to about 693,000 with 4-bit weights, 11% more room for requests (the cache itself stays in BF16 here).

Quality: small losses down to 4 bits, a cliff below

Two measures, telling slightly different stories.

GSM8K barely moves. Every 8-bit and 4-bit format, GGUF Q4_K_M included, scored between 92.2% and 93.3%, against 93.1% for BF16. How much of that is real? One run gives a hint: a configuration that, as it turned out, left the model in BF16 (see below) scored 92.65%, six problems below the BF16 run with identical weights. On 1,319 problems, differences of half a point are within run-to-run noise. Only two results stand clearly outside it: GGUF Q3_K_M (91.7%) and Q2_K (89.1%, four points down).

Perplexity separates the formats. It measures how well the model predicts real text, token by token, and moves with every change to the weights. The 8-bit formats add 0.03 to 0.11 to BF16's 9.72. The 4-bit formats add 0.3 to 0.6, and the method matters: round-to-nearest INT4 is the worst at 10.29, GPTQ (10.05) and AWQ (10.18) do better, partly because they also store a zero point per group (round to nearest with a zero point scores 10.12, measured afterwards for GPTQ, Explained and Run), and NF4 (9.99) does best among the weight-only 4-bit formats. GGUF Q4_K_M, really 4.9 bits per weight with sensitive tensors kept in 6 bits, sits with the 8-bit formats at 9.82. Q2_K, at 3.2 bits, adds more than a full point.

Which measure to trust depends on the use: GSM8K says whether this model still solves these problems; perplexity says how far the model has drifted, which tends to show up first on harder or longer tasks. A later piece in this series covers evaluations that catch quantization damage.

Prompts and batches: where activations matter

Reading a long prompt is the opposite regime from decoding one token: thousands of tokens go through each weight at once, and the GPU is limited by arithmetic, not memory. Here the split is between formats that quantize activations and those that do not:

• W8A8 and W4A4 are faster. FP8 and MXFP8 read the prompt in about 1.2 seconds, NVFP4 in 0.88, against 1.8 for BF16: their matrix multiplications run on the GPU's FP8 and FP4 tensor cores.
• Weight-only 4-bit is slower than BF16. AWQ, GPTQ, round-to-nearest INT4 (all through vLLM's Marlin kernel) and NF4 took 2.2 to 2.4 seconds. They convert the 4-bit weights back to 16 bits before multiplying, extra work that decoding hides and prompt reading does not.

The same split shows with 32 concurrent requests: NVFP4 reached 711 tokens per second, FP8 about 520, the weight-only INT4 formats about 600, NF4 422, BF16 332. Weight-only formats win when one request at a time is decoding; formats with quantized activations win as soon as the GPU is busy.

What did not run

Four of the seventeen configurations failed, and they are as informative as the rest:

• INT8 W8A8 (made with llm-compressor, SmoothQuant + GPTQ): vLLM refused it: "Int8 not supported on SM121. Use FP8 quantization instead, or run on older arch (SM < 100)." On this Blackwell GPU, 8-bit means FP8 (Which GPU Computes Which Format).
• MXFP4 (made with llm-compressor): the FlashInfer kernel library reported "mm_fp4 does not support backend 'cute-dsl' with capability 121". A format the GPU supports in hardware still needs a kernel for that exact GPU.
• 2-bit GPTQ (a public AutoRound checkpoint): "Unsupported quantization config: bits=2". vLLM 0.31.0 does not run this 2-bit configuration.
• INT8 weight-only, quantized at load time: vLLM accepted --quantization int8_per_channel_weight_only, started, and served, but its log said "Quantized 0 layers": the model ran in BF16. The flag gave no error. Check the startup log for what was actually quantized.

How to choose

• Memory-bound, few requests at a time: 4-bit weights (AWQ, GPTQ, NVFP4) decode about three times faster than BF16 for about a point of GSM8K at most on this model. Prefer a checkpoint with a zero point; a well-calibrated GPTQ improves on it a little more.
• Busy server or long prompts: formats that quantize activations (FP8, NVFP4 on Blackwell) are faster everywhere, including prompt reading; weight-only 4-bit is slower than BF16 there.
• Safest: FP8 is within noise of BF16 on GSM8K and adds 0.1 to perplexity, at 1.6 to 1.8 times the decode speed.
• Around 4 bits per weight and below: quality starts to drop measurably (Q3_K_M, really 4.0 bits: 91.7%; Q2_K, 3.2 bits: 89.1% on GSM8K).
• Before trusting a format on your GPU: start it and read the log. Four of seventeen did not do what their name said on this one.

The pieces that follow explain why the methods differ: how post-training quantization handles outliers, and how GPTQ and AWQ choose their integers.