The Case for FP8

The DeepSeek-V3 paper reports that pre-training a 671 B-parameter model on 14.8 T tokens cost about 2.788 million H800 GPU-hours. Without FP8, the same run would have cost roughly twice that — and more importantly, it would not have fit on the 2048 H800s the team had access to. FP8 is not a micro-optimisation. It is the difference between training a frontier model and watching one train somewhere else. This section makes the case for FP8 before the next two sections explain why naïve FP8 fails and how DeepSeek made it work.

The Real Problem: 671 B Parameters Don't Fit

A frontier-scale dense or MoE model carries an enormous amount of state through every training step. For each parameter you store the weight itself, the gradient, two Adam optimiser moments, and a slice of the forward-pass activations needed for backprop. With $N = 671 \cdot 10^{9}$ parameters in pure FP32 you are looking at roughly $16 N = 10.7 \, \text{TB}$ of weight-plus- optimiser state alone, before activations and the KV cache that long-context training demands.

An H800 GPU has 80 GB of HBM. To hold the FP32 state of a 671 B model you would need $10.7 \, \text{TB} / 80 \, \text{GB} \approx 134$ GPUs just to park the weights, before you spend a single FLOP on training. Add activations, gradients in transit, and all-to-all buffers for expert parallelism and the floor climbs to many hundreds of GPUs — at which point the network becomes the bottleneck and tensor cores starve.

The old fix was BF16 weights with FP32 master copies (Micikevicius et al. 2018). That cuts forward-pass memory in half. The H100's FP8 tensor cores cut it in half again — and double matmul throughput at the same time. For DeepSeek-V3's 2048-GPU cluster, FP8 is the only path that closes both the memory budget and the compute budget at once.

Intuition: Pay for Bits That Earn Their Keep

A floating-point number is a budget. Each bit you spend either buys you range (the orders of magnitude you can represent — the exponent) or precision (how finely you can split each order of magnitude — the mantissa). FP32 buys both lavishly. FP8 forces a brutal trade-off: with only eight bits total, you cannot afford both wide range and fine resolution.

Think of the bits as ruler markings. FP32 is a long ruler with $2^{23} \approx 8 \cdot 10^{6}$ tick marks between consecutive powers of two — so fine you cannot see the ticks. BF16 keeps the same length but uses only $2^7 = 128$ ticks per octave. FP8 E4M3 has only $2^3 = 8$ ticks per octave and a much shorter ruler — about six decades wide. Any value outside that ruler becomes zero or saturates; any value inside the ruler gets snapped to the nearest of the eight ticks in its octave.

The bet of FP8 training is that, with the right scaling, almost all the numbers a transformer block needs to multiply land inside FP8's range, and that the few-percent rounding error per multiply is recoverable by a higher-precision accumulator. The rest of this chapter is the engineering required to make that bet true.

The Anatomy of a Float: FP32, BF16, FP16, FP8

Every IEEE-style float encodes a value as:

$x = (-1)^{s} \cdot 2^{e - \text{bias}} \cdot \left( 1 + \frac{m}{2^{M}} \right)$

with $s \in \{0, 1\}$ the sign, $e$ a non-negative integer stored in $E$ exponent bits, and $m$ the mantissa stored in $M$ bits. The bias $\text{bias} = 2^{E-1} - 1$ centres the exponent range around 1 so that both very small and very large numbers are representable.

The dynamic range of the format — the ratio of max to min representable normal — is roughly:

$\text{range} \approx 2^{2^{E} - 1}$

and the relative resolution within a single octave — the smallest fractional change you can resolve — is:

$\epsilon = 2^{-M}$

Substitute the four formats and watch the trade-off snap into focus:

Read the rows. BF16 has FP32's range with 4× worse precision — which is fine for forward pass because the rounding error per multiply is absorbed by an FP32 accumulator. FP16 has FP32's precision-ish but a much narrower range — which is why FP16 training requires loss scaling to keep gradients from underflowing.

FP8 E4M3 has even narrower range AND much coarser precision: it gives up about $10^{4}$× of dynamic range vs BF16 and $16$× of precision. FP8 E5M2 trades a precision bit back for a range bit — recovering BF16's range but with only 4 distinct values per octave. Neither format can survive on its own without scaling; the engineering question is what kind of scaling.

The two scaling regimes

The simplest fix is per-tensor scaling: multiply the whole tensor by a constant before casting to FP8 so that its amax lands at FP8's max value. One scale per tensor, one extra FP32 scalar to store. Works fine for weights, where the distribution is stable across steps.

It fails for activations. After a non-linearity, a single outlier can be 20× the bulk of the distribution — pin the amax to 448 and the bulk gets squashed into a handful of FP8 buckets, with relative error north of 50%. The DeepSeek-V3 paper's answer is per-block scaling (a separate scale per 128-element block of the tensor) plus higher-precision accumulation — the topic of section 10.3. For now, the case for FP8 stops at: the hardware is there, the memory savings are there, the throughput is there, and the scaling problem is solvable.

Manual Numerical Walkthrough

Two examples, calculated by hand: casting a handful of values to FP8 E4M3, and sizing the memory savings on a 671 B-parameter MoE like DeepSeek-V3.

Visualizing Range, Precision, and Memory

Three widgets, each isolating one of FP8's trade-offs. The first compares the representable range of FP32, BF16, FP16, and FP8 against the magnitudes that actually appear in a transformer block. The second is a bit-level explorer — flip bits, watch the value change, see how the sign-exponent-mantissa split builds the number. The third sizes the whole-model memory bill for every realistic precision recipe.

Flip a single mantissa bit on FP8 E4M3 and watch the value jump by ~12.5% — that is $\epsilon = 2^{-3}$ in action. Flip the same bit on FP32 and the value moves by $2^{-23} \approx 10^{-7}$. This is the precision cost of the format, in one click.

Plain Python: Simulating FP8 Quantisation

Before we call into a Hopper tensor core, let us build the cast by hand. The function below is a faithful emulation of what the hardware does to every number: round-to-nearest-even, saturate at ±448, flush sub-subnormals to zero. Then we use it to simulate an FP8 matmul against an FP32 reference and measure the rounding error a transformer block has to live with.

A quantised to FP8 E4M3

B quantised to FP8 E4M3

matmul, accumulator in FP32

1import math
2import numpy as np
3
4# FP8 E4M3: 1 sign bit, 4 exponent bits, 3 mantissa bits.
5# Bias = 7 (so unbiased exponent runs roughly -6 .. +8).
6E_BITS, M_BITS, BIAS = 4, 3, 7
7MAX_NORMAL  = (2 - 2**-M_BITS) * 2**(2**E_BITS - 1 - BIAS - 1)   # ≈ 448
8MIN_NORMAL  = 2**(1 - BIAS)                                       # ≈ 0.0156
9MIN_SUBNORM = 2**(1 - BIAS - M_BITS)                              # ≈ 0.00195
10
11def quantize_e4m3(x):
12    """Round-to-nearest-even cast from FP32 to FP8 E4M3."""
13    if x == 0:
14        return 0.0
15    sign = math.copysign(1.0, x)
16    ax = abs(x)
17
18    # Above max: saturate (DeepSeek uses saturated cast, not inf).
19    if ax >= MAX_NORMAL:
20        return sign * MAX_NORMAL
21
22    # Below smallest subnormal: flush to zero.
23    if ax < MIN_SUBNORM / 2:
24        return 0.0
25
26    # Decompose into exponent and significand.
27    exp = math.floor(math.log2(ax))
28    exp = max(exp, 1 - BIAS)                  # subnormal floor
29    frac = ax / 2**exp                        # in [1, 2) for normals
30    scaled = frac * 2**M_BITS                 # in [8, 16) for normals
31    rounded = round(scaled)                   # round-to-nearest-even
32    if rounded == 2**(M_BITS + 1):            # carry into next exponent
33        rounded //= 2
34        exp += 1
35    return sign * rounded * 2**(exp - M_BITS)
36
37def fp8_matmul(A, B):
38    """Simulate an H100 FP8 GEMM: quantise inputs, multiply in FP32, return FP32."""
39    qA = np.vectorize(quantize_e4m3)(A)
40    qB = np.vectorize(quantize_e4m3)(B)
41    return qA @ qB                            # accumulator stays FP32
42
43# Quick sanity check: cast a few canonical values.
44for x in [1.0, 0.5, 0.1, 1e-3, 1e-5, 100.0, 500.0, -3.14]:
45    q = quantize_e4m3(x)
46    rel_err = abs(q - x) / max(abs(x), 1e-30)
47    print(f"x = {x:>10.5g}   q(x) = {q:>10.5g}   rel.err = {rel_err:6.2%}")
48
49# Matmul fidelity: random Gaussian inputs, compare to FP32 reference.
50np.random.seed(0)
51A = np.random.randn(128, 128).astype(np.float32)
52B = np.random.randn(128, 128).astype(np.float32)
53ref     = A @ B
54fp8_out = fp8_matmul(A, B)
55print(f"\nMatmul max abs error : {np.abs(ref - fp8_out).max():.3f}")
56print(f"Matmul mean rel error: {(np.abs(ref - fp8_out) / np.abs(ref).clip(1e-6)).mean():.2%}")

PyTorch: Real FP8 Matmul on Hopper

Now the same idea on real silicon. PyTorch 2.1+ exposes Hopper's FP8 tensor cores through $\texttt{torch.\_scaled\_mm}$ — give it FP8 inputs, an FP32 accumulator, and a pair of scales, and you get back the matmul at roughly twice BF16's throughput. This is the kernel that sits inside DeepSeek-V3's forward and backward passes; later sections will wrap it in per-block scaling, but the raw call is here.

weights, shape (M, K), std ≈ 0.02

activations, shape (K, N), std ≈ 1.0

FP8 E4M3, shape (M, K)

FP8 E4M3, shape (K, N)

result, shape (M, N), BF16 dtype

1import torch
2
3# Hopper (H100/H200) and Blackwell GPUs expose two FP8 dtypes natively.
4# E4M3: wider mantissa, lower range — used for activations & weights.
5# E5M2: wider range, lower mantissa — used for gradients (need range).
6e4m3 = torch.float8_e4m3fn
7e5m2 = torch.float8_e5m2
8
9device = "cuda"
10M, K, N = 4096, 4096, 4096
11
12# 1) Allocate FP32 tensors with realistic transformer-block scale.
13A_fp32 = torch.randn(M, K, device=device) * 0.02      # weights ~ N(0, 0.02)
14B_fp32 = torch.randn(K, N, device=device) * 1.0       # activations ~ N(0, 1)
15
16# 2) Per-tensor amax scale: map the largest abs value to FP8 max (448 for E4M3).
17amax_A = A_fp32.abs().amax()
18amax_B = B_fp32.abs().amax()
19scale_A = 448.0 / amax_A.clamp(min=1e-12)
20scale_B = 448.0 / amax_B.clamp(min=1e-12)
21
22# 3) Quantise: multiply by scale, cast to FP8.
23A_fp8 = (A_fp32 * scale_A).to(e4m3)
24B_fp8 = (B_fp32 * scale_B).to(e4m3)
25
26# 4) FP8 matmul on Hopper tensor cores. Accumulator stays FP32.
27out_fp32 = torch._scaled_mm(
28    A_fp8, B_fp8,
29    scale_a=1.0 / scale_A,                # inverse scales applied to output
30    scale_b=1.0 / scale_B,
31    out_dtype=torch.bfloat16,
32)
33
34# 5) Reference and diff.
35ref = (A_fp32 @ B_fp32).to(torch.bfloat16)
36err = (out_fp32 - ref).abs()
37print(f"FP8 GEMM  : {M}x{K} @ {K}x{N}")
38print(f"max abs err: {err.max().item():.4f}")
39print(f"mean rel err: {(err / ref.abs().clamp(min=1e-3)).mean().item():.2%}")
40
41# 6) Memory footprint comparison.
42def mb(t): return t.element_size() * t.numel() / 1024**2
43print(f"\nFP32 weights : {mb(A_fp32):8.2f} MB")
44print(f"FP8  weights : {mb(A_fp8):8.2f} MB  ({mb(A_fp32) / mb(A_fp8):.1f}x smaller)")

At Massive Scale: Why DeepSeek-V3 Bet the Run on FP8

DeepSeek-V3 is the largest open-weight model trained primarily in FP8 to date — 671 B total parameters, 37 B activated per token, 14.8 T training tokens. The paper reports the run on 2048 H800 GPUs for about two months, at a quoted cost of 5.576 million USD in compute. Without FP8, neither the memory budget nor the time budget closes. Three constraints pushed the team to FP8.

1. Memory ceiling. 2048 H800s give 160 TB of total HBM. Holding 671 B parameters in BF16 weights + BF16 gradients + FP32 master weights + FP32 Adam moments costs $(2 + 2 + 4 + 8) N = 16 N \approx 10.7 \, \text{TB}$ — 6.7% of total HBM, before activations, KV cache, or expert routing buffers. With FP8 weights and FP8 gradients that drops to $14 N \approx 9.4 \, \text{TB}$, and DeepSeek further shards optimiser state across DP ranks so the per-GPU footprint fits with comfortable margin.
2. Throughput ceiling. H800s deliver 989 BF16 TFLOPS and 1979 FP8 TFLOPS per GPU. The training run requires $6 N D \approx 6 \cdot 10^{25}$ FLOPs. At the realised MFU (~40% on H800 with FP8) the run takes $\approx 50$ days. In BF16 at the same MFU you double that — well past the team's available compute window.
3. Communication ceiling. Expert parallelism requires all-to-all every layer. The volume of those collectives is proportional to the activation tensor size, which is proportional to the precision width. FP8 halves the wire bytes relative to BF16, taking the network from co-bottleneck to comfortably non-binding — a key reason DeepSeek can fit fine-grained MoE inside their fabric budget.
4. Iteration speed. Most of the run cost is not the headline pre-training pass — it is the dozens of ablations, recipe sweeps, and recovery restarts that precede and surround it. A 2× compute speed-up at the same loss is a 2× research velocity gain across the entire programme. Frontier labs notice.

The shape of the FP8 forward pass at scale

Inside a single DeepSeek-V3 transformer block, here is what is actually FP8 and what is not:

Roughly 60–70% of the FLOPs in a transformer block are GEMMs that can run in FP8. Softmax, normalisation, and optimiser updates stay in higher precision — a hybrid recipe that captures most of the speed-up while leaving the numerically sensitive stages alone.

Engineering Reality: What FP8 Buys, What It Costs

FP8 is not free. The four engineering costs you sign up for the moment you turn it on:

1. Hardware lock-in. FP8 is only fast on Hopper (H100/H200) and Blackwell (B100/B200/GB200). Ampere (A100) and earlier generations have no FP8 tensor cores — an FP8 cast on A100 saves memory but not compute. Cross-cluster training (mixed hardware) becomes painful.
2. Numerical fragility. A single un-scaled activation outlier can saturate a whole block's GEMM, and gradient distributions shift across training — what was a safe scale at step 1k can be wrong at step 100k. Sections 10.2 and 10.3 are the cost of this fragility: per-block scaling, dynamic scale tracking, and tile-level outlier handling.
3. Debugging is worse. "The loss spiked at step 47k" in BF16 usually means "a data shard was bad". In FP8 it can also mean "the scale for layer 38 drifted" or "an outlier in expert 47's input overflowed". New failure mode taxonomy, new monitoring infra.
4. Framework support is uneven. PyTorch supports$\texttt{torch.\_scaled\_mm}$ but not (yet) FP8 autograd everywhere. Most production FP8 training stacks (TransformerEngine, DeepSeek's in-house kernels, Megatron-LM-FP8) reach into custom CUDA. Off-the-shelf is not yet enough.

Against those four costs, the benefits compound:

The deepest lesson. Every order of magnitude of scaling has historically required a precision step-down. Single GPUs used FP32. The first multi-GPU runs (Megatron, GPT-3) moved to FP16/BF16 with master weights. Frontier scale (DeepSeek-V3, Grok-2, internal labs at OpenAI and Anthropic) is moving to FP8. The next scale-up almost certainly involves FP4 or block-floating- point formats — the same engineering programme, one bit narrower. The lesson of this chapter is not "use FP8", it is learn to engineer training around precision, because the next format is already on the roadmap.