FP16 vs. BF16: Same Size, Different Trade

Open almost any model card and you will find a line like torch_dtype: bfloat16. Start vLLM with --dtype half and the same model runs in FP16. Both are 16 bits per number, the same memory, the same speed on a modern GPU. They are not the same format, and the difference decides whether a model trains, and sometimes whether it runs.

This piece puts the two side by side and measures what changes on a real model. It builds on How a Computer Stores a Real Number: a float is a sign, an exponent, and a mantissa.

Same 16 bits, split two ways

FP16 is the IEEE standard's half-precision format. BF16, "brain floating point", was designed at Google Brain. Its trick is to be the top half of an FP32 number: same sign bit, same 8 exponent bits, and the first 7 mantissa bits. Converting FP32 to BF16 means rounding away the low 16 bits; converting back means appending 16 zeros.

Range: where FP16 runs out

On a log scale the difference is stark. BF16's normal numbers span about 76 orders of magnitude, FP16's about 9 (12 counting subnormals). At the top, FP16 stops at 65,504: anything larger becomes infinity, and infinity turns into NaN at the first subtraction or multiplication by zero. At the bottom, FP16 loses precision below 0.000061 (the subnormal range) and reaches zero below about 0.00000003.

Precision: what the extra 3 bits buy

FP16 has 3 more mantissa bits, so its steps are 8 times finer. Does that matter for a model's output? I ran Qwen3-0.6B on the same 80-token paragraph three times, in FP32 (the reference), BF16, and FP16, and compared the next-token probabilities at every position:

KL divergence measures how far one probability distribution is from another; 0 means identical. FP16 lands about 50 times closer to FP32 than BF16 does. At three positions out of 80, BF16's rounding was enough to change the most likely token. Both errors are small, but the direction is clear: when the numbers fit, FP16 is the more faithful format.

Why training chose BF16 anyway

Training produces numbers that inference does not: gradients, and they are often tiny. A large share can fall below FP16's smallest values and become zero, so the weights they should update never move.

The 2017 paper that made FP16 training practical ("Mixed Precision Training", Micikevicius et al.) fixed this with two techniques: keep an FP32 master copy of the weights, and multiply the loss by a large factor before the backward pass, so the gradients shift up into FP16's range ("loss scaling"). PyTorch still ships this as GradScaler: it starts at a scale of 65,536 (2¹⁶), halves it and skips the step whenever a gradient overflows to infinity, and doubles it again after 2,000 clean steps. It works, but it is one more moving part, and a run can still blow up.

BF16 removes the problem at the source. With FP32's exponent, gradients that fit in FP32 fit in BF16. Google's TPU team put it this way: "neural networks are far more sensitive to the size of the exponent than that of the mantissa", and BF16 "comes close to being a drop-in replacement for FP32" without loss scaling. A 2019 study by Intel and Facebook researchers trained image, speech, language, and recommendation models in BF16 and reached the same results as FP32 "with no changes to hyper-parameters". Once GPUs added BF16 tensor cores, training moved to it, and models were released in it.

What it means for inference

A released model has finished training. No gradients, no loss scaling: only weights and activations. So the question for inference is narrower: do this model's numbers fit in FP16? I measured it on Qwen3-8B, which is released in BF16.

The weights fit. Of 8.19 billion weights, none exceeds FP16's range (the largest is 34). Every weight between 0.000061 and 65,504 converts exactly, because BF16's 8 significant bits fit inside FP16's 11. 38 million weights (0.47%) are smaller than that and become subnormal in FP16, losing some bits; 19,276 become zero.

The activations fit, with room to spare.

The first six layers stay below about 100. At layer 7 a handful of values jump to 12,096 and stay near 12,000 to the end. This is a known phenomenon called massive activations: a few values thousands of times larger than the rest, which "largely stay constant regardless of the input" and act as fixed biases inside the model (Sun et al., 2024). The peak, 12,544, is about a fifth of 65,504.

The output does not change. With the whole model in FP16, the three test prompts produced no infinity or NaN anywhere, and the greedy 30-token continuations were identical to BF16's, word for word.

So for this model, FP16 inference is safe, and if anything slightly more precise. That is not a rule. The Hugging Face Transformers documentation walks through a T5 model whose feed-forward output reached 62,700, then crossed 65,504 one operation later and produced infinities. It also warns that "NaN loss often occurs when a model is pretrained in bf16 and then it is used with fp16". Code can overflow too: a hard-coded attention mask of −1e9 becomes −infinity in FP16. The margin depends on the model, so measure it.

In practice

Speed is the same. Both formats take 2 bytes per value, so they move the same amount of memory, and recent tensor cores compute them at the same rate (NVIDIA lists 1,979 TFLOPS for both on the H100, with sparsity). The only reason to pick one over the other is numerical.

Serving. vLLM's --dtype auto (the default) uses the dtype in the model's config, BF16 for most current models. --dtype half forces FP16; vLLM allows it but logs a warning when it casts between BF16 and FP16. Prefer the released format unless you have a reason, such as a GPU without BF16 support (NVIDIA added BF16 tensor cores with Ampere).

Checking a model. To know whether a BF16 model is safe in FP16, do what this article did: run a few representative prompts with forward hooks that record the largest absolute value per layer, and compare with 65,504. The measurement script for this article is short Transformers code. Also compare a few outputs token by token.

What to take away

• Same size, different split: FP16 has more precision (10 mantissa bits), BF16 has FP32's range (8 exponent bits).
• Training moved to BF16 because gradients need range; FP16 training needs loss scaling.
• For inference, FP16 is fine when every activation stays under 65,504. On Qwen3-8B the peak is 12,544, and FP16 tracks FP32 about 50 times more closely than BF16.
• Some models overflow in FP16. Measure the activation peaks before switching.

The next piece goes down to 8 bits, where the trade is so tight that there are two FP8 formats: E4M3 for precision and E5M2 for range.