Quantization, Mapped: Weights, Activations, KV Cache

A quantized checkpoint's name often hides three choices. "W4A16" means weights in 4 bits, activations in 16. "W8A8" means both in 8. "FP8 KV cache" means the attention cache is stored in FP8. Each choice quantizes a different kind of number, saves a different resource, and carries a different risk. This piece maps the three before the rest of the series goes into methods.

Three kinds of numbers

• Weights are the model's parameters: the matrices in every projection. They never change while serving, and every decoding step reads all of them.
• Activations are the vectors flowing between layers, one per token. They are different for every input.
• The KV cache holds the keys and values of every past token of every request, so attention does not recompute them. It grows with each token generated.

Weights: memory and decoding speed

Quantizing the weights shrinks the model in memory, and that alone speeds up decoding. Generating one token for one request reads every weight once and does little arithmetic with each, so decoding is limited by memory bandwidth (The Roofline for LLMs). Fewer bytes, faster tokens: on a GB10, Qwen3-8B's weights went from 15.27 GiB in BF16 to 5.98 GiB in NVFP4, and single-request decoding went from 13.6 to 36.1 tokens per second, 2.7 times faster for 2.6 times fewer bytes (NVFP4 on Blackwell).

Weight-only quantization (W4A16, W8A16) gets this gain without touching the math: the kernel turns the weights back into 16 bits just before multiplying. Some weights are usually left alone. The embeddings and the output layer often stay in BF16: for Qwen3-8B, with a 151,936-token vocabulary and separate input and output matrices, they are 2.3 GiB on their own, which is why a "4-bit" Qwen3-8B is 6 GiB and not 4. Norms stay in higher precision too (Where Low Precision Bites).

The risk is moderate and well understood: weights have outliers, so they need fine-grained scales (Integers for AI: INT4 needs groups of 128 or 32), and methods like GPTQ and AWQ pick the integers to protect the layer's outputs.

Activations: compute

Weight-only quantization saves bytes but not arithmetic. When many tokens are processed at once, reading a long prompt or decoding for many requests, the GPU is limited by computation instead, and the only way to go faster is to multiply in a smaller format. That needs both operands quantized: the weights and the activations. On the same GB10, NVFP4 weights and activations (W4A4) read an 8,000-token prompt in 871 ms against 1,735 ms for BF16, which a weight-only format would not do.

This is the riskiest of the three. Activations are different for every input, so their scales are computed on the fly, often per token. And they have outliers far worse than weights: Qwen3-8B carries "massive activations" of about 12,000 from layer 7 on (FP16 vs. BF16). That is why the format matters more here: on Qwen3-8B's activations, FP8 E4M3 has a quarter of INT8's error with one scale per tensor (FP8), and NVFP4's fine block scales bring 4-bit activation error down to 3.7% (NVFP4). Methods like SmoothQuant and rotations exist to tame activation outliers before quantizing them.

KV cache: memory per token

The KV cache costs a fixed amount per token: for Qwen3-8B, 36 layers × 8 key-value heads × 128 dimensions × 2 (keys and values) × 2 bytes = 144 KiB per token in BF16. That adds up fast:

With one long request, the weights dominate. With 32 requests of 8,000 tokens, the cache needs 36 GiB in BF16, more than twice the BF16 weights. Storing it in FP8 halves it: in vLLM, --kv-cache-dtype fp8 (E4M3; fp8_e5m2 is the alternative). On the GB10, the NVFP4 checkpoint that also stored its KV cache in FP8 had room for 1.35 million cached tokens against 668,000 without, twice as many requests at once.

The risk sits between weights and activations: keys and values are activations stored for later, and attention reads all of them for every new token. A later piece in this series measures what FP8 and smaller caches cost.

Reading the names

The KV cache setting is independent of the checkpoint: you can store the cache in FP8 with BF16 weights, or keep it in BF16 with 4-bit weights.

What comes next

The rest of this series takes each part of this map in turn: what one model costs in size, speed, and quality in a dozen formats, measured (How Much Does Each Bit Cost?); how post-training quantization handles outliers; GPTQ, AWQ, SmoothQuant, and rotations; quantization-aware training; the KV cache; GGUF; mixture-of-experts models; and how to measure the damage.