TF32 and Accumulation: What Tensor Cores Really Compute In

Ask PyTorch for a matrix multiplication of two FP32 tensors on a recent NVIDIA GPU, and you may or may not get FP32. With one flag set, the GPU's tensor cores round your inputs to fewer bits, multiply them, and add the results at full precision. The answer comes back as an FP32 tensor either way. This article opens up the multiplication to show which precision is used where, and measures what each choice costs in accuracy and speed.

Three precisions in one matmul

Every element of a matrix product is a dot product: multiply pairs of numbers, then add the results up.

C[i][j] = A[i][0]·B[0][j] + A[i][1]·B[1][j] + ... + A[i][K-1]·B[K-1][j]

That single line involves three precisions:

1. Inputs: the format A and B are read in (FP32, TF32, BF16, FP16, FP8...).
2. Products: each multiplication. Multiplying two numbers with p significant bits gives an exact result with 2p bits, so tensor cores can compute products of small inputs exactly or nearly so.
3. Accumulator: the running sum. This is where most of the rounding happens, once per added term, K times per output.

When a kernel is described as "BF16 with FP32 accumulate", it means BF16 inputs and an FP32 running sum. That is the default for BF16 and FP16 matmuls on NVIDIA tensor cores in PyTorch.

TF32: FP32's range with FP16's precision

TF32 ("TensorFloat-32") is not a storage format. Your tensors stay FP32 in memory. When TF32 is allowed, the tensor core reads each FP32 input, keeps its sign, its 8 exponent bits, and the top 10 mantissa bits, and drops the other 13. That is 19 bits: the range of FP32 with the precision of FP16. The products are then summed in FP32. PyTorch's documentation describes it as "rounding input data to have 10 bits of mantissa, and accumulating results with FP32 precision, maintaining FP32 dynamic range".

The point is speed: FP32 matmuls can use tensor cores, with no change to your code or your data. The cost is precision on the inputs.

PyTorch has changed its mind about the default. TF32 for matmuls was on by default in PyTorch 1.7 to 1.11 and has been off since 1.12 (convolutions in cuDNN still default to TF32). You turn it on with one line:

torch.backends.cuda.matmul.fp32_precision = "tf32"   # newer API; "ieee" for full FP32
torch.set_float32_matmul_precision("high")          # older equivalent

If torch.compile has warned you that "TensorFloat32 tensor cores for float32 matrix multiplication" are "available but not enabled", this is the switch it means.

Why the accumulator matters

A simple experiment shows what a low-precision accumulator does. Start at 0 and add 1.0, over and over, keeping the running sum in a given format:

At 2,048, the gap between neighboring FP16 numbers becomes 2 (the first article in this series shows why gaps double at every power of two). 2,048 + 1 lands exactly halfway between 2,048 and 2,050, rounds to the even one, and the sum stays at 2,048 forever. Every one added after that is lost. A dot product with thousands of terms has the same problem in a milder form: as the sum grows, small terms are rounded away.

To see it on real matmuls, I multiplied random matrices with each input/accumulator combination and compared the result with the same multiplication in FP64, for dot products from 256 to 65,536 terms:

Two different things are going on:

• With an FP32 accumulator, the error does not grow with K. It is set by the rounding of the inputs: about 0.03% for TF32 and FP16 (10 mantissa bits), about 0.3% for BF16 (7 bits). Adding more terms does not make it worse, because the FP32 sum loses almost nothing.
• With an FP16 accumulator, the error grows with K: from 0.07% at 256 terms to 0.28% at 65,536, four times worse, and about eight times the error of the same FP16 inputs with an FP32 sum.

This is why "FP32 accumulate" is the default and why it matters. Reduced-precision inputs cost a fixed, predictable error. A reduced-precision accumulator costs an error that grows with the size of the problem.

PyTorch exposes the accumulator too. torch.backends.cuda.matmul.allow_fp16_accumulation = True asks cuBLAS for full FP16 accumulation, which "certain GPUs" run faster "at the cost of numerical precision and greater likelihood of overflow" (PyTorch docs). Separate flags allow some intermediate reductions in reduced precision; for BF16 matmuls that one is on by default.

What each choice costs in speed

The same 4,096 × 4,096 × 4,096 matmul, timed on one GPU (an NVIDIA GB10, PyTorch 2.11 with cuBLAS, median of 20 runs):

TF32 more than doubles FP32's speed for an error of 0.03%, without touching your data. 16-bit inputs are five times faster than FP32: a matmul this size is limited by arithmetic, and tensor cores process 16-bit inputs at a higher rate than TF32. And on this GPU, accumulating in FP16 buys no speed at all over FP32 accumulation: it only adds error. The ratios differ from one GPU to the next (PyTorch's documentation reports TF32 about 7 times faster than FP32 on the A100), so measure on yours; the errors do not change.

FP8: when the accumulator is not FP32 even by default

The same question came back with FP8. The DeepSeek-V3 team, training in FP8 on NVIDIA H800 GPUs (a Hopper GPU), found that the tensor cores' FP8 matmul does not accumulate in full FP32: "the accumulation precision of FP8 GEMM on NVIDIA H800 GPUs is limited to retaining around 14 bits". In their test on random matrices with K = 4,096, this gave a maximum relative error of nearly 2%.

Their fix was the idea of this article in hardware terms: let the tensor core sum 128 products at its limited precision, then copy the partial sum to FP32 registers on the regular CUDA cores and add it there. The error stays bounded because no low-precision sum ever runs longer than 128 terms. The report calls the limitation a "hardware deficiency" and asks future chips to "adopt higher precision". The report also notes that the limited precision "is still the default option in a few FP8 frameworks". The FP8 piece of this series returns to it.

In practice

• Training or fine-tuning in FP32? Allow TF32 unless you have a reason not to (scientific code that needs full FP32). The input rounding is about 0.03%, fixed, and the accumulation stays FP32.
• Serving in BF16 or FP16? The standard matmul paths (cuBLAS by default) accumulate in FP32, so you mostly pay the input rounding of the format. The BF16 reduced-precision reduction flag is the exception to know about.
• Writing or choosing a kernel? Check the accumulator, not only the input format. "FP16 matmul" says nothing about it; "FP16 with FP32 accumulate" does.
• Long reductions (large K, long sequences in attention, big sums in normalization layers) are where a low-precision accumulator shows.

What to take away

• A matmul has three precisions: inputs, products, and the accumulator.
• TF32 = FP32's sign and exponent with a 10-bit mantissa, used only inside tensor cores; off by default for matmuls in PyTorch since 1.12.
• With an FP32 accumulator, the error is set by the inputs and stays flat as sums get longer. With an FP16 accumulator it grows: an FP16 running sum of ones stops at 2,048.
• FP8 on Hopper accumulates in about 14 bits; DeepSeek-V3 promoted partial sums to FP32 every 128 products.

The next piece looks at the two 8-bit floating-point formats, E4M3 and E5M2, and why there are two.