NVFP4: Two Levels of Scale

To rebuild one weight from an NVFP4 checkpoint, you multiply three numbers: a 4-bit value, the 8-bit scale of its block of 16, and a 32-bit scale shared by the whole tensor. NVIDIA designed the format for Blackwell's tensor cores. The microscaling piece showed that MXFP4, the open standard, loses accuracy mostly to its power-of-two scale. This piece shows what NVFP4 changes, and measures what each change buys.

Three numbers per value

• The element: FP4 E2M1, the same as MXFP4. Sixteen bit patterns: ±0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6.
• The block scale: one per 16 values (MXFP4: 32), stored in FP8 E4M3 (MXFP4: E8M0, a power of two). It is set so the block's largest value lands on 6.
• The tensor scale: one FP32 number for the whole tensor, chosen so that all the block scales fit in E4M3's range.

That is 4 bits per value plus 8 bits per 16 values: 4.5 bits per value, against 4.25 for MXFP4 (the tensor scale is one number for millions of values and adds nothing measurable).

Try it on 64 real weights from Qwen3-8B, and make one of them an outlier to see how a block scale contains the damage:

What each part buys

To see where NVFP4's accuracy comes from, I changed the scale design one step at a time and measured the error on all of Qwen3-8B's linear-layer weights and on the inputs of every linear layer over three prompts:

Reading it from top to bottom:

1. Blocks do most of the work. One scale per tensor leaves FP4 useless on weights (68% error); any block scale brings it to about 10 to 12%.
2. A fine scale matters more than a small block. Going from MXFP4's power-of-two scale to an exact scale on the same blocks of 32 cuts the activation error from 11.3% to 4.4%. Going from blocks of 32 to blocks of 16 with exact scales then gains another point (to 3.4%). With a power-of-two scale, even blocks of 16 stay at 11%.
3. E4M3 is nearly as good as exact. NVFP4's 8-bit block scale gives 9.5% and 3.7%, within a few tenths of exact FP32 scales.

Why there is a tensor scale

E4M3 is a small format: its values run from 0.002 (2⁻⁹) to 448. A block scale is the block's largest value divided by 6, so it must land in that range, and real tensors do not cooperate at either end:

• Weights are small. The typical block of 32 Qwen3-8B weights in the integer piece's widget tops out at 0.069, which needs a block scale of about 0.012: below E4M3's smallest normal value (0.0156), in its imprecise subnormal range.
• Activations can be huge. Qwen3-8B has "massive activations" around 12,000 (FP16 vs. BF16). A block containing one needs a scale of about 2,000, far above E4M3's 448. The scale clamps, and the large value is clipped.

The FP32 tensor scale shifts the whole tensor so that its block scales land inside E4M3's range. Remove it and store the block scales directly, and the measurement shows exactly this: weights get a little worse (9.5% to 9.9%), activations collapse (3.7% to 36.8%).

What it costs, and where it runs

NVFP4 spends a quarter of a bit more per value than MXFP4 (4.5 against 4.25) and needs one extra multiplication per tensor. In exchange, on Qwen3-8B its activation error is about a third of MXFP4's. That matters most when activations are quantized too (W4A4), which is what lets the tensor cores run at their FP4 rate.

NVIDIA's Blackwell generation computes NVFP4 natively (the hardware piece of this series lists the rates per GPU), and on other GPUs engines fall back to slower paths. How to serve an NVFP4 checkpoint with vLLM, and what it changes in speed and accuracy on a real model, is the subject of NVFP4 on Blackwell.

What to take away

• NVFP4 = FP4 E2M1 values × an E4M3 scale per 16 values × an FP32 scale per tensor: 4.5 bits per value.
• Its accuracy over MXFP4 comes mostly from the finer block scale, a little from the smaller blocks.
• The tensor scale keeps the block scales inside E4M3's range; without it, the activation error on Qwen3-8B rises from 3.7% to 37%.