TF32 is NVIDIA’s hybrid floating-point format used inside Tensor Cores (Ampere and later). It keeps the range of FP32 but uses only a 10-bit mantissa (same as FP16).
It is not a storage format. Values stay in FP32 memory; the GPU rounds them to TF32 only for the multiply inside matrix operations, then accumulates back in full FP32.
Introduced by NVIDIA with the Ampere architecture (A100 GPU, 2020), TF32 is a compute format designed specifically for deep-learning matrix multiplications.
Deep learning needs a wide dynamic range (like FP32) but can tolerate reduced precision in the mantissa. TF32 gives you exactly that combination.
| Format | Sign | Exponent | Mantissa | Total bits | Range | Approx. decimal digits |
|---|---|---|---|---|---|---|
| FP32 | 1 | 8 | 23 | 32 | ~1e-38 … 3e38 | ~7.2 |
| TF32 | 1 | 8 | 10 | 19 (internal) | ~1e-38 … 3e38 | ~3.1 |
| FP16 | 1 | 5 | 10 | 16 | ~6e-5 … 6e4 | ~3.1 |
| BF16 | 1 | 8 | 7 | 16 | ~1e-38 … 3e38 | ~2.1 |
Notice that TF32 has the same exponent as FP32 (full range) and the same mantissa as FP16.
This is why you get a large speedup (often 8× on A100 vs pure FP32) with only a small accuracy impact for most neural networks.
The graph above shows the distribution of representable TF32 values in [0, 1]. Because TF32 has only 10 mantissa bits, there are far fewer unique values than FP32, so the points are much more spaced out — especially visible on the linear scale.
Before Ampere, training in FP32 was accurate but slow. FP16 was fast but had a small dynamic range, requiring loss scaling. BF16 fixed the range problem but reduced precision further.
NVIDIA created TF32 as a “best of both worlds” format for Tensor Cores:
torch.backends.cuda.matmul.allow_tf32.