FP8: E4M3, E5M2, and Why There Are Two

An FP8 checkpoint of Qwen3-8B is half the size of the BF16 original, and on GPUs with FP8 tensor cores it computes at twice the rate (NVIDIA lists 3,958 FP8 TFLOPS on the H100 against 1,979 for BF16, both with sparsity). But "FP8" is not one format. It is two, with different bit splits, and both need a helper to work at all. This piece explains both, and measures them on a real model against the other 8-bit option, INT8.

Two encodings

With 8 bits, the split between exponent and mantissa matters a lot, so the standard defines two:

E5M2 follows the usual IEEE rules, with infinity and NaN. E4M3 bends them to gain range: it gives up infinity and keeps a single NaN pattern, which frees enough codes to reach 448 instead of 240. The paper that defined both formats (Micikevicius et al., 2022) says how to use them: "E4M3 for weight and activation tensors, and E5M2 for gradient tensors". Gradients span a wider range and tolerate coarser steps; weights and activations need the extra mantissa bit. For inference, that means E4M3 almost everywhere.

FP8 needs a scale

448 is a small maximum, and Qwen3-8B's weights are small numbers, typically a few hundredths, down in the part of E4M3 where it has few values. Cast them to E4M3 directly and the result is poor: of 6.95 billion weights, 352 million (5%) become zero and 2.8 billion (40%) land in the subnormal range below 0.016, where E4M3 has only 7 values in all.

So FP8 always comes with a scale, as integers do: divide the tensor by a scale so that its largest value lands near 448, store the FP8 values and the scale, multiply back when computing. The FP8 paper compares it to the loss scaling of FP16 training, and leaves the choice of scale to software: "the general idea is to choose a scaling factor such that the maximum magnitude in the tensor becomes close to the maximum" of the format.

FP8 against INT8, on a real model

Both formats are 8 bits with a scale. Which is more accurate? I quantized all of Qwen3-8B's linear-layer weights each way, with one scale per tensor and one per row:

Two results:

• FP8 does not care who shares the scale. E4M3 gives 2.65% error with one scale for a whole matrix and 2.64% with one per row. A float's steps grow with the value, so the error stays a roughly fixed fraction of each weight, whatever the scale. INT8's evenly spaced steps are the opposite: one scale per tensor gives 7.6%, one per row 1%.
• With a fine enough scale, INT8 is more precise. 8 bits of evenly spaced integers resolve a row of weights better than 3 mantissa bits do. E5M2, with only 2 mantissa bits, has twice E4M3's error.

Activations tell a different story. They change with every input, so their scales are computed on the fly, often one per tensor or one per token, and they contain outliers, like the "massive activations" of about 12,000 measured in FP16 vs. BF16. The same measurement on the inputs of every linear layer, over three prompts:

Here FP8 wins: E4M3's error is about a quarter of INT8's with one scale per tensor, and a third with one per token. A few huge values set the scale, INT8's evenly spaced steps become coarse for everything else, and FP8's relative precision does not. That robustness is a large part of FP8's appeal when activations are quantized too, even though INT8 is more precise on the weights alone.

FP8 in practice

Checkpoints. Qwen's own Qwen3-8B-FP8 stores weights in E4M3 with one scale per block of 128 × 128 weights, and computes activation scales on the fly ("dynamic"). Red Hat AI's Qwen3-8B-FP8-dynamic uses one scale per output channel for weights and one per token for activations. Red Hat's leaves the output layer (lm_head) in BF16. Either loads in vLLM like any model.

Without a checkpoint. vLLM can quantize a BF16 model to FP8 while loading it, with --quantization fp8, and offers variants by scale granularity (per tensor, per channel, per 128 block). Since the error barely depends on granularity, the simplest one is usually fine, which is a large part of FP8's appeal: no calibration data, no special tooling.

The KV cache. --kv-cache-dtype fp8 stores the attention cache in E4M3 (fp8_e5m2 is the other option), halving its memory. The quantization series measures what it does to quality.

Hardware. FP8 tensor cores arrived with NVIDIA's Hopper and Ada generations and are in every NVIDIA data-center GPU since. On GPUs without FP8 hardware, vLLM falls back to its Marlin kernel for "weight-only FP8 quantization" (compute capability 7.5 or higher): the weights stay FP8 in memory and the math runs in 16 bits, which saves memory but not compute. The hardware piece of this series lists which GPU does what. And recall from TF32 and Accumulation that Hopper's FP8 matmuls accumulate with about 14 bits, which DeepSeek worked around by promoting partial sums to FP32 every 128 products.

What to take away

• FP8 is two formats: E4M3 (448 max, 3 mantissa bits) for weights and activations, E5M2 (57,344 max, 2 bits) for gradients.
• FP8 needs a scale: cast directly, 40% of Qwen3-8B's weights would land in E4M3's subnormal range.
• FP8's error is relative, so it barely depends on how many values share a scale (2.6% on Qwen3-8B's weights either way); INT8 needs a scale per row to reach 1%.
• Real FP8 checkpoints use E4M3 with block or per-channel scales for weights and dynamic scales for activations.

The next piece goes below 8 bits, where one scale per tensor is no longer enough even for floats: the microscaling formats MXFP8, MXFP6, and MXFP4.