Integers for AI: INT8, INT4, Scales, and Zero Points

Open the files of a 4-bit checkpoint such as Qwen3-8B-AWQ and you will not find a single real number for the weights. You will find packed integers between 0 and 15, and, every 128 of them, a 16-bit number called a scale and a 4-bit number called a zero point. That is enough to rebuild every weight to within about 10%. This piece explains how, and measures what each setting costs on every weight of a real model.

It is the integer counterpart of How a Computer Stores a Real Number: floats carry their own exponent in every number; integers share one scale across many.

An integer and a scale

The rule is a single line:

value = (integer − zero_point) × scale

With 8 bits you have 256 integers, with 4 bits only 16. The scale says how far apart they are in real units, and the zero point says which integer stands for 0. To store a weight, divide by the scale, round to the nearest integer, add the zero point.

This grid is uniform: every step is the same size, unlike floats, whose steps double at every power of two. A float format puts many values near zero and few far away; an integer format spreads them evenly between the smallest and largest value the scale can reach. That is why the scale matters so much: it decides where those evenly spaced steps go.

Symmetric or asymmetric

There are two ways to pick the scale.

• Symmetric: the integers run from −8 to 7 (for INT4), 0 stands for 0, and the scale is the largest absolute value divided by 7. Simple, and the zero point is free (it is always 0).
• Asymmetric: the integers run from 0 to 15, the scale is (max − min) / 15, and a zero point shifts the grid so that it starts at the minimum. It costs a few extra bits per group to store the zero point.

Weights are not always centered on zero. In this real row of Qwen3-8B, they run from −0.467 to 0.809. Symmetric quantization has to cover −0.809 to 0.809 and leaves 4 of its 16 codes below anything the row contains. Asymmetric quantization spends all 16 on the range that exists. At 8 bits the difference is small; at 4 bits, a quarter of the codes is a lot.

Who shares a scale

The other choice is how many weights share one scale:

• Per tensor: one scale for a whole weight matrix (for Qwen3-8B, up to 50 million weights).
• Per channel (per row): one scale for each output row of the matrix, thousands of weights.
• Per group: one scale for every 128 or 32 consecutive weights in a row.

Smaller groups mean more scales to store, but each scale fits its weights more closely. The reason this matters is outliers. Weights in a trained model are mostly small, with a few much larger ones. In Qwen3-8B, the largest weight of a down_proj matrix is typically 45 times the standard deviation of that matrix. One outlier sets the scale for everything that shares it, so the steps become large, and the small weights next to it round to the same few integers, often to 0.

Try it on real weights: a typical group of 32, and the group in the same matrix where one weight (0.42) towers over its neighbors (around 0.01). Switch to INT4 and widen the scale to the row or the tensor, and watch the small weights collapse to zero.

Measured on every weight of Qwen3-8B

To see the effect on a whole model, I quantized all 252 linear-layer matrices of Qwen3-8B (6.95 billion weights) with each setting and measured how far the result is from the original:

The error is the size of the difference relative to the size of the weights (root-mean-square), with the simplest method, round to nearest. Three things stand out:

• INT8 is forgiving. With one scale per row, the weights are within about 1%. That is why INT8 weights with one scale per row are a safe choice.
• INT4 is not. With one scale per tensor, the error is 87%: the weights are essentially gone. Per row is still 19%. It takes groups of 128 or 32 to bring it to 10 to 12%.
• The zero point helps most where bits are scarce: about 2 points at INT4, a tenth of a point at INT8.

This is exactly what 4-bit checkpoints do. Qwen3-8B-AWQ and Red Hat AI's Qwen3-8B-quantized.w4a16 both use 4-bit integers in groups of 128, with a zero point. A 16-bit scale every 128 weights adds 0.125 bit per weight, so "4-bit" models are really about 4.2 bits. They also do not round to nearest: GPTQ and AWQ choose the integers to keep each layer's outputs close to the original, not the weights themselves, which the quantization series covers. And weight error is not model quality: what a 10% weight error costs on real tasks is measured in the same series.

Weights only, or activations too

Checkpoint names often say which numbers are integers:

• W4A16 (weight-only): weights are INT4, activations stay BF16. At run time, the kernel turns the integers back into BF16 just before multiplying. It saves memory and memory bandwidth, which is what limits decoding, but the math itself runs in BF16.
• W8A8: weights and activations are both INT8, and the multiplication runs on the GPU's integer units. NVIDIA lists 3,958 trillion INT8 operations per second on the H100, twice its FP16 rate (both with sparsity). Activations change with every input, so their scales are usually computed on the fly, one per token.

Activations bring their own outlier problem, often worse than weights (FP16 vs. BF16 measured activations of 12,000 in Qwen3-8B). The SmoothQuant piece in the quantization series shows how W8A8 handles it.

What to take away

• An integer format stores value = (integer − zero point) × scale. The steps are uniform.
• Symmetric fixes 0 at integer 0; asymmetric adds a zero point to use all codes on the real range.
• The fewer weights share a scale, the smaller the error, because outliers stretch the grid for everything that shares their scale.
• Measured on Qwen3-8B: INT8 is fine per row (1%); INT4 needs groups of 128 or 32 (10 to 12%), and real 4-bit checkpoints use groups of 128 with a zero point.

The next piece returns to floating point, at 8 bits: FP8 E4M3 and E5M2, and why there are two.