How a Computer Stores a Real Number: Sign, Exponent, Mantissa
What the bits of a floating-point number mean, why range and precision trade against each other, with an interactive bit explorer.
formats
9 posts
FP16 vs. BF16: Same Size, Different Trade
Why training moved from FP16 to BF16, and why a BF16 model may still run fine in FP16: the same 16 bits split differently, measured on Qwen3-8B.
TF32 and Accumulation: What Tensor Cores Really Compute In
Inputs, products, and accumulators: the precision a matrix multiplication really uses, and why FP32 accumulation still matters.
Integers for AI: INT8, INT4, Scales, and Zero Points
How integers represent real weights: scales, zero points, per-tensor, per-channel, and per-group, measured on every weight of Qwen3-8B.
FP8: E4M3, E5M2, and Why There Are Two
The two FP8 formats, what each is for, and how FP8 is scaled, measured on Qwen3-8B's weights and activations against INT8.
Microscaling: MXFP8, MXFP6, MXFP4 and Block Scales
The OCP microscaling formats: 32 values sharing one power-of-two scale, what MXFP8, MXFP6, and MXFP4 cost in accuracy on a real model, and why the scale rule matters.
NVFP4: Two Levels of Scale
How NVFP4 differs from MXFP4: 16-value blocks, FP8 block scales, and a per-tensor scale, with what each part buys measured on Qwen3-8B.
Which GPU Computes Which Format: Ampere to Rubin, MI300 to MI355
Which number formats each data-center GPU computes natively, at what rate, from the vendors' own pages: A100 to Rubin, MI300X to MI355X.
Where Low Precision Bites: Rounding, Overflow, and Accumulation
Five places where low precision breaks real computations, each reproduced on Qwen3-8B's own tensors, and the fix every serious implementation applies.