Post-Training Quantization: Rounding, Calibration, and the Outlier Problem

Model cards for quantized checkpoints often say something like "calibrated on 512 samples". It is a strange thing to read about a conversion: why would turning 16-bit numbers into 4-bit ones need data at all? Because the goal is not to keep each weight close to its original value. It is to keep the model's outputs close to the original model's, and that depends on the inputs it sees. This piece explains the gap, and measures the outliers that make it hard.

Round to nearest, and what it misses

Post-training quantization (PTQ) means quantizing a model after it has been trained, without training it further. The simplest method is round to nearest (RTN): choose a scale for each group of weights (Integers for AI), divide, round each weight to the nearest integer. It needs no data and takes seconds.

RTN minimizes the error of each weight taken alone. But a weight is never used alone. A linear layer computes dot products, Σ wᵢ·xᵢ, so the error that reaches the next layer is Σ (q(wᵢ) − wᵢ)·xᵢ: each weight's rounding error multiplied by the input it meets. A small error on a weight that meets a large input matters more than a large error on a weight that meets a near-zero input, and errors can add up or cancel along the sum. RTN is blind to both, because it never looks at the inputs.

Outliers: where the inputs are large

The inputs are not uniform. Some are much larger than others, and those outliers are what make quantization hard, in two places.

In the weights. Each weight matrix has a few values far from the rest: in Qwen3-8B's down_proj matrices, the largest weight is typically 45 times the standard deviation. Fine-grained scales (groups of 128 or 32) contain the damage (Integers for AI).

In the activations. Activations are worse, and they are the inputs that multiply the weights. I ran Qwen3-8B on 2,048 tokens of Wikipedia text and recorded, for each input channel of two layers, the largest value it took:

In the input of the attention projection (q_proj), a typical channel peaks at 0.60. Five of the 4,096 channels exceed ten times that, and the largest, channel 3828, reaches 26 times, and it is among the ten largest values of every token. This is the "highly systematic emergent features" that the LLM.int8() paper (Dettmers et al., 2022) identified: a few fixed dimensions that are large all the time. In the MLP's intermediate activations (the input of down_proj), the picture differs: outliers are larger (up to 43 times the median channel, 77 channels above 10 times) but they move from token to token; the top channels are among a token's largest values on at most 22% of tokens.

The two kinds behave differently under quantization. LLM.int8()'s trick is to keep the few outlier dimensions in 16 bits and quantize the rest. Applied to these activations with INT8 and one scale per token:

Setting aside 5 channels of q_proj's input cuts the error by almost two-thirds; doing the same for down_proj's input barely helps, because its outliers are not in fixed channels. Persistent channel outliers can be handled by treating those channels specially (in 16 bits, or by rescaling them); outliers that move with the token need something else, such as finer scales per token or per block (NVFP4), or rotations that spread them out (a later piece).

Calibration: learning where the inputs are large

Calibration means running a small set of representative inputs, a few hundred samples of text, through the model and recording what the layers see. Each method then uses those statistics differently:

• llama.cpp's importance matrix records the average squared activation of each input channel and uses it to weight the rounding error: weights that meet large inputs are rounded more carefully (GGUF Quant Types).
• AWQ (Lin et al., 2023) observes that "protecting only 1% salient weights can greatly reduce quantization error", that the salient channels are found from "the activation distribution, not weights", and protects them by scaling those channels up before quantizing (and the matching activations down), a change that leaves the layer's math unchanged.
• GPTQ (Frantar et al., 2022) quantizes a weight matrix one column at a time and, using second-order statistics of the calibration activations, adjusts the columns not yet quantized to compensate for each rounding error, so that the errors cancel in the output instead of adding up.
• SmoothQuant (Xiao et al., 2022), for W8A8, "migrates the quantization difficulty from activations to weights": it divides the outlier activation channels by a factor and multiplies the matching weights by it, since "weights are easy to quantize while activations are not".

All four use the same data, the activations, for the same reason: the error that matters is the error in the layer's output.

Measured: what calibration buys

On Qwen3-8B with 4-bit integer weights in groups of 128, served by vLLM (How Much Does Each Bit Cost?):

Symmetric round to nearest raises perplexity from 9.72 to 10.29. The GPTQ and AWQ checkpoints also use a zero point, which on its own lowers 4-bit weight error (Integers for AI), so the fair comparison is round to nearest with a zero point: 10.12. Against it, GPTQ (10.05) still gains, AWQ (10.18) does not on this test. The zero point removes about 30% of the loss by itself; calibration, done well, removes some more. On GSM8K, all of them stay within about a point of BF16, the size of run-to-run noise on this test, so the differences show in perplexity first.

The cost of calibration

Calibration data is not free of risk. The method fits the data it sees: text unlike your use can steer the rounding toward the wrong channels. Unsloth reports that calibrating on Wikipedia-like text makes quantized models look better on Wikipedia-based evaluations than they are, and AWQ's authors present its reliance on activation statistics alone, without reconstruction, as a way to avoid "overfitting the calibration set". Calibrate on data that looks like what the model will serve.

What to take away

• Round to nearest minimizes each weight's error; what matters is the error in the layer's output, weighted by the inputs.
• Calibration runs a few hundred samples through the model to learn which inputs are large.
• Activation outliers come in two kinds: fixed channels (in Qwen3-8B's attention input, one channel is 26× typical on every token) and token-specific spikes (in the MLP, up to 43×).
• On Qwen3-8B, a zero point alone removes about 30% of round to nearest's perplexity loss; the published GPTQ checkpoint removes more (40%), the AWQ checkpoint less. Compare methods at the same scheme.

The next pieces open the two most common methods: GPTQ, explained and run, and AWQ.