Where Low Precision Bites: Rounding, Overflow, and Accumulation

The matrix multiplications in a model are the part everyone worries about when lowering precision, and they are the part that is best protected: tensor cores add their products in FP32 by default, and quantized formats come with carefully chosen scales. The failures happen elsewhere, in the small operations around them: a softmax, a normalization, an optimizer step, a running sum, a cast. This piece reproduces five of them on Qwen3-8B's own tensors, gives the threshold for each, and shows the fix that real implementations use.

1. exp() overflows: the naive softmax

A softmax turns scores into probabilities: exp(xᵢ) / Σ exp(xⱼ). In FP16, exp(x) overflows to infinity as soon as x exceeds 11.09, because e¹¹·⁰⁹ ≈ 65,504, FP16's largest number. Infinity divided by infinity is NaN.

Language models produce scores well above 11. On three prompts, Qwen3-8B's final logits peaked at 19.25, 28.9, and 34.0. A naive softmax over them in FP16 overflows on 502, 2,016, and 7,629 vocabulary entries respectively, and every output probability comes back NaN.

The fix is to subtract the largest score before exponentiating: exp(xᵢ − max) / Σ exp(xⱼ − max). Mathematically it is the same function; numerically every exponent is now at most 0, so exp() is at most 1 and cannot overflow. With it, FP16 gives the right answer (top probability 0.9668 against 0.9669 in FP64 on the third prompt). BF16, whose exp() only overflows above 88.7, survives even the naive version here, but its probabilities are off in the third digit.

Where to look: hand-written softmax or log-softmax in sampling code, custom attention kernels, and anything computing exp() of scores in 16 bits.

2. x² overflows: a normalization in 16 bits

RMSNorm, the normalization in most current LLMs, divides each token's vector by the root of its mean square. It squares every value first, and in FP16 the square of anything above 255.9 (√65,504) overflows.

Qwen3-8B has massive activations: from layer 7 on, a few values reach about 12,000 (FP16 vs. BF16), concentrated on a few tokens. I took the input of layer 10's normalization on three prompts (20 tokens) and computed the RMSNorm entirely in FP16. For 3 of the 20 tokens, a square overflowed, the mean became infinity, its inverse square root became 0, and the whole normalized vector came out as zeros. No NaN, no error: the token's representation silently vanished.

The fix is to compute the reduction in FP32 and convert back at the end. Both Transformers and vLLM do it. Transformers' Qwen3RMSNorm starts with hidden_states.to(torch.float32) before computing the variance; vLLM's reference rms_norm does x.to(torch.float32) before x.pow(2).mean(...). The upcast also helps precision when nothing overflows: in BF16, it halves the error of the normalized vector (0.16% against 0.28% median, compared with an FP32 reference).

Where to look: any norm, variance, or sum of squares written in 16 bits, in custom fused kernels especially.

3. Small updates round away: why master weights stay in FP32

A training step changes each weight by a small amount: learning rate times a gradient-derived step. If the weight is stored in BF16, the update must be larger than half the gap between the weight and its neighbor, or it rounds back to the same value and is lost.

To see how often that happens, I added an update of size u to every weight of one Qwen3-8B matrix (50 million weights, layers.10.mlp.down_proj) and counted the weights that did not change:

Adam-style optimizers move each weight by roughly the learning rate, and learning rates around 10⁻⁵ are common in fine-tuning. At u = 10⁻⁵, 88% of the weights stored in BF16 do not move at all. FP16, with 3 more mantissa bits, loses the same share at updates ten times smaller. FP32 loses none.

The fix has been standard since mixed-precision training was introduced (Micikevicius et al., 2017): keep a master copy of the weights in FP32, apply the updates to it, and use a 16-bit copy for the forward and backward passes. Training code that keeps weights only in BF16 has to deal with this some other way, or lose most small updates.

Where to look: training or fine-tuning code that keeps weights and optimizer state only in 16 bits; long training runs where the learning rate decays to small values.

4. Long sums stall: accumulation

The same rounding hits any running sum. Start at 0 and add 1 in FP16: the sum stops at 2,048, where the gap between FP16 numbers becomes 2 (TF32 and Accumulation). Every dot product in a matrix multiplication is such a sum, which is why tensor cores accumulate in FP32 by default, and why a matmul with an FP16 accumulator measured four times more error at 65,536 terms than at 256.

FP8 brought the problem back: on Hopper, FP8 matrix multiplications accumulate with about 14 bits, and the DeepSeek-V3 team measured a maximum relative error of nearly 2% at K = 4,096 before promoting partial sums to FP32 every 128 products.

Where to look: reductions over long dimensions (sums over vocabulary, sequence, or hidden size) in custom kernels; FP8 GEMM settings that trade accumulation precision for speed.

5. Casting without a scale: clipped or NaN

Converting a tensor to FP8 without scaling it first assumes its values fit in the format. They often do not: E4M3 stops at 448, and Qwen3-8B's activations reach 12,544. What happens then depends on the library:

PyTorch's cast (measured on CPU) saturates to 448: the value is silently clipped, by a factor of 28 for 12,544. NumPy's ml_dtypes returns NaN instead, as shown in How a Computer Stores a Real Number. Neither is correct; the first is harder to notice.

The fix is the one every quantization scheme in this series uses: compute a scale from the tensor's largest value (per tensor, per row, or per block) and divide by it before casting. For activations, whose range changes with every input, that means dynamic scales computed at run time, or a calibration that has seen the outliers.

Where to look: any .to(torch.float8_e4m3fn) without a preceding scale; static activation scales calibrated on data that never triggered the model's massive activations.

A checklist

When a model misbehaves in low precision (NaN, garbage output, or a quiet loss of accuracy), check these first:

• Softmax and log-softmax: is the maximum subtracted before exp()?
• Norms, variances, sums of squares: are they computed in FP32?
• Optimizer: are master weights FP32? Are tiny learning rates being rounded away?
• Accumulators: are long reductions and matmul accumulations in FP32?
• Casts to FP8 or FP4: is there a scale, and does it see the outliers?
• Overflow is not always loud: an RMSNorm in FP16 produced zeros, and an FP8 cast clipped, without a single NaN.

This closes the number formats series. The quantization series picks up from here: which parts of a model to quantize, with which method, and what it costs in quality, measured.