Fine-Grained Quantisation

One activation channel out of four thousand can hold a value a hundred times larger than every other channel. If you give the whole tensor a single FP8 scale, that one channel decides what every other number is allowed to look like — and the rest get crushed to zero. DeepSeek-V3's answer is brutally simple: stop using one scale. Give every 1×128 strip of activations and every 128×128 patch of weights its own FP32 multiplier. That is fine-grained quantisation, and it is the only reason FP8 training works at 671 B parameters.

The Real Problem: One Outlier Burns Every Other Number

FP8 E4M3 — the activation format used by DeepSeek-V3, NVIDIA H100 Transformer Engine, and every modern FP8 training stack — has exactly $2^{8} = 256$ representable values. Its maximum magnitude is $448$. To store a tensor of FP32 values $x$ in FP8, the standard recipe is symmetric scaling:

$x_{\text{fp8}} = \text{round}\!\left(\frac{x}{s}\right), \quad s = \frac{\max(|x|)}{448}$

The scale $s$ is a single FP32 number that pushes the largest absolute value in the tensor exactly to the FP8 ceiling. Everything smaller than $s$ rounds to zero. So far, so good — for well-behaved tensors.

Transformer activations are not well-behaved. Dettmers et al. (2022, LLM.int8()) measured them carefully and found something that the architecture papers never talk about: a small number of specific channels — sometimes just 0.1% of all channels — carry activation magnitudes 10× to 100× larger than the median. These outlier channels are not transient. They appear at the same indices across thousands of inputs. They grow more prominent as models get larger. By 6.7 B parameters, they dominate the per-tensor amax.

Zeroing out 80% of activations does not produce a NaN. It does not crash. It produces a model that does not learn — gradients are computed from these mostly-zero tensors, the optimiser pretends everything is normal, and after several thousand steps the loss curve simply refuses to come down. Engineering teams have wasted whole training runs chasing this exact bug. The first attempts at FP8 training in 2022–2023 all hit it.

And outliers are not the only failure mode. Even without a catastrophic cell, the tail of a Gaussian-shaped activation distribution wastes most of FP8's tiny dynamic range whenever one tile happens to contain a 4-σ value and the next contains none. Per-tensor scaling forces both tiles to share the same ruler, and the ruler has only 8 mantissa levels per binade. Two tiles, one ruler, eight levels — most values land between the same two FP8 codes.

Intuition: Many Local Rulers Beat One Global Ruler

Imagine you are asked to measure the heights of every object in a warehouse — pencils, boxes, forklifts, an industrial crane — and write each measurement on a card with only three digits of precision. You have a single ruler and you have to choose its length up front.

A short ruler — say 1 metre — gives you millimetre precision on pencils and boxes, but the crane does not fit. Your only option is to write "1000+" for the crane and lose the information.

A long ruler — say 30 metres — fits the crane (30 m written as 29.7) but now everything else has 100-mm precision. The pencil and the box are indistinguishable. Per-tensor FP8 is the long-ruler strategy: one global scale chosen to fit the biggest value, and every smaller value loses its resolution.

The fine-grained answer is obvious once you see it: do not use one ruler. Issue a separate ruler to every workstation. The pencil bench gets a 30 cm ruler with millimetre marks. The pallet area gets a 3 m ruler with centimetre marks. The crane gets its own giant ruler. Each ruler still has only three digits, but each ruler is calibrated to the local range. Globally, you measure everything with maximum local resolution. Locally, you give up nothing.

The picture extends to weights. A weight matrix in a transformer FFN has its own outliers — usually tied to specific input channels. DeepSeek-V3 quantises weights in 128×128 patches: each patch acts as a small "neighbourhood" with its own ruler. The patch is larger than the activation tile because weights are accessed in a different pattern by the GEMM kernel (this is the topic of §10.4), but the philosophy is identical.

The Block-Wise Quantisation Equations

Let $X \in \mathbb{R}^{R \times C}$ be the tensor we want to quantise. Partition it into rectangular blocks of shape $B_r \times B_c$. Call the block at row-block $i$, column-block $j$ the matrix $X^{(i,j)} \in \mathbb{R}^{B_r \times B_c}$. For each block we compute one FP32 scale:

$s^{(i,j)} = \frac{\max_{p,q} |X^{(i,j)}_{p,q}|}{F_{\max}}$

where $F_{\max} = 448$ for E4M3. The block is then quantised:

$Q^{(i,j)}_{p,q} = \text{round}\!\left(\frac{X^{(i,j)}_{p,q}}{s^{(i,j)}}\right) \in [-F_{\max}, +F_{\max}]$

The reconstruction (dequantised) value is $\hat{X}^{(i,j)}_{p,q} = s^{(i,j)} \cdot Q^{(i,j)}_{p,q}$. The block-local quantisation error $\epsilon^{(i,j)} = \hat{X}^{(i,j)} - X^{(i,j)}$ is bounded by half a quantisation step:

$|\epsilon^{(i,j)}_{p,q}| \leq \frac{s^{(i,j)}}{2}$

Read that bound carefully. The error in a block is proportional to that block's scale, which is proportional to that block's amax. A block with no outliers has a small amax → small scale → tiny error. A block with one outlier has a large amax → large scale → large error everywhere in that block. The damage is confined to the block that contains the outlier. That is the entire mathematical content of fine-grained quantisation.

Memory accounting

For a tensor of shape $(R, C)$ with blocks of shape $(B_r, B_c)$, the storage is:

$\text{bytes} = \underbrace{R \cdot C}_{\text{FP8 payload}} + \underbrace{4 \cdot \lceil R / B_r \rceil \cdot \lceil C / B_c \rceil}_{\text{FP32 scales}}$

The overhead ratio is $4 / (B_r \cdot B_c)$. At $B_r = B_c = 128$ (DeepSeek weight policy) this is $4 / 16384 \approx 0.024\%$ — negligible. At $(B_r, B_c) = (1, 128)$ (DeepSeek activation policy) this is $4 / 128 \approx 3.1\%$ — small. At per- element $(1, 1)$ it would be 400% — worse than just keeping FP32. The block size is where the trade-off lives.

Why per-token and per-channel are NOT enough

Some early FP8 systems used per-row scaling for activations: one scale per token. That works when the outlier pattern is per-row. But outlier channels — fixed column indices that misbehave in every row — defeat per-row scaling: every row has that one big column. A different early strategy used per-channel scaling: one scale per column. That defeats outlier tokens, but creates a problem for the matmul kernel — the contraction axis for an activation × weight GEMM is the channel axis, and you cannot accumulate at FP8 if the scale varies along the contraction axis without breaking the GEMM's inner-loop scaling invariant. DeepSeek-V3's 1×128 tile splits the difference: one scale per (token, 128-channel-group). Outlier channels are absorbed into their own 128-group; outlier tokens are absorbed into their own row; the contraction-axis tiling cleanly matches the GEMM's K-block, which lets the FP8 kernel broadcast the scale across the K-loop in registers.

Manual Numerical Walkthrough

Eight numbers. Two strategies. One outlier. All arithmetic by hand so you can verify every step.

Visualizing Where the Damage Goes

The widget below shows a 16×32 activation tensor with three outlier channels and one catastrophic single-cell spike. Switch between the four scaling strategies and watch the dashed block boundaries shrink. The error heatmap reveals where the damage accumulates: with per-tensor scaling almost every cell is contaminated; with 1×128 tile scaling only the tile containing the catastrophic cell is hurt; the rest of the tensor regains near-FP8 precision.

Plain Python: Per-Tensor vs Block-Wise

Pure Python, no NumPy, no PyTorch. The whole machinery of FP8 block-wise quantisation in under 40 lines. Read this once and the DeepSeek paper's quantisation diagrams stop looking like magic — they are this loop, vectorised on a GPU.

1import math
2import random
3
4FP8_MAX = 448.0   # E4M3 maximum representable magnitude
5
6def quantise_block(values):
7    """Symmetric block quantisation: one FP32 scale per block."""
8    amax = max(abs(v) for v in values)
9    scale = amax / FP8_MAX if amax > 0 else 1e-30
10    # Round to the FP8 integer grid (8 mantissa levels per binade in E4M3;
11    # we model that here as integer rounding after scaling — exact behaviour
12    # depends on the format, but the headline is identical).
13    q = [max(-FP8_MAX, min(FP8_MAX, round(v / scale))) for v in values]
14    recon = [scale * x for x in q]
15    err = [r - v for r, v in zip(recon, values)]
16    return scale, recon, err
17
18def quantise(tensor, block):
19    """Quantise a 2-D tensor (list of lists) with rectangular blocks."""
20    R, C = len(tensor), len(tensor[0])
21    br, bc = block
22    recon = [[0.0] * C for _ in range(R)]
23    scales = []
24    for r0 in range(0, R, br):
25        for c0 in range(0, C, bc):
26            vals = [tensor[r0 + i][c0 + j]
27                    for i in range(br) for j in range(bc)]
28            s, rec, _ = quantise_block(vals)
29            scales.append(((r0, c0), s))
30            k = 0
31            for i in range(br):
32                for j in range(bc):
33                    recon[r0 + i][c0 + j] = rec[k]
34                    k += 1
35    return recon, scales
36
37def rmse(a, b):
38    n = len(a) * len(a[0])
39    s = sum((a[r][c] - b[r][c]) ** 2 for r in range(len(a)) for c in range(len(a[0])))
40    return math.sqrt(s / n)
41
42# Build a 16 x 32 activation tensor: mostly N(0,1), with two outlier
43# channels and a single catastrophic cell — mirroring real LLM activations.
44random.seed(0)
45x = [[random.gauss(0, 1) for _ in range(32)] for _ in range(16)]
46for r in range(16):
47    x[r][5]  = 80 + random.random() * 40    # outlier channel
48    x[r][18] = 60 + random.random() * 30    # outlier channel
49x[3][12] = 220.0                            # catastrophic single cell
50
51per_tensor, s_pt = quantise(x, (16, 32))    # 1 scale total
52per_row,    s_pr = quantise(x, (1, 32))     # 16 scales (one per token)
53tile_1x8,   s_t  = quantise(x, (1, 8))      # 64 scales (DeepSeek-style)
54
55print(f"per-tensor   RMSE = {rmse(x, per_tensor):.3f}   ({len(s_pt)} scales)")
56print(f"per-row      RMSE = {rmse(x, per_row):.3f}   ({len(s_pr)} scales)")
57print(f"1x8 tile     RMSE = {rmse(x, tile_1x8):.3f}   ({len(s_t)} scales)")

PyTorch: DeepSeek-Style Tile and Block Scaling

Same logic, vectorised across a real-sized activation tensor with the exact (1, 128) and (128, 128) granularities DeepSeek-V3 uses. The reshape-and-permute trick collapses the per-block loop into two fused PyTorch ops — no Python iteration, no per-tile kernel launch. On an H100 this entire round-trip on a 1024×4096 tensor takes well under a millisecond.

1import torch
2
3FP8_MAX = 448.0  # E4M3
4
5def quantise_fp8_blockwise(x: torch.Tensor, block: tuple[int, int]) -> tuple[torch.Tensor, torch.Tensor]:
6    """Symmetric block-wise FP8 simulation: returns (dequantised tensor, FP32 scales)."""
7    R, C = x.shape
8    br, bc = block
9    assert R % br == 0 and C % bc == 0, "tensor must be divisible by block size"
10
11    # 1. Reshape into (R/br, br, C/bc, bc) so each (br, bc) tile is contiguous.
12    tiles = x.view(R // br, br, C // bc, bc)            # (nR, br, nC, bc)
13    tiles = tiles.permute(0, 2, 1, 3).contiguous()      # (nR, nC, br, bc)
14
15    # 2. amax per tile -> FP32 scale per tile.
16    amax = tiles.abs().amax(dim=(-1, -2), keepdim=True) # (nR, nC, 1, 1)
17    scale = (amax / FP8_MAX).clamp(min=1e-12)           # avoid div-by-zero
18
19    # 3. Quantise + clamp to FP8 range, then dequantise.
20    q = (tiles / scale).clamp(-FP8_MAX, FP8_MAX).round()
21    recon = q * scale                                   # (nR, nC, br, bc)
22
23    # 4. Restore the original (R, C) layout.
24    recon = recon.permute(0, 2, 1, 3).reshape(R, C)
25    return recon, scale.squeeze(-1).squeeze(-1)         # (nR, nC) scales
26
27# Build a realistic 1024 x 4096 activation tensor with outlier channels.
28torch.manual_seed(0)
29A = torch.randn(1024, 4096)
30A[:, 137] *= 30.0     # outlier channel 1
31A[:, 901] *= 50.0     # outlier channel 2
32A[42, 2719] = 220.0   # single catastrophic cell
33
34# DeepSeek-V3 activation policy: 1 x 128 tile.
35A_recon_act, scales_act = quantise_fp8_blockwise(A, block=(1, 128))
36
37# DeepSeek-V3 weight policy: 128 x 128 block (simulated on the same matrix
38# to illustrate the granularity — in practice this is applied to W, not A).
39A_recon_w, scales_w = quantise_fp8_blockwise(A, block=(128, 128))
40
41# Per-tensor baseline (the failure mode).
42A_recon_pt, scales_pt = quantise_fp8_blockwise(A, block=(1024, 4096))
43
44def snr_db(x, x_hat):
45    return 10 * torch.log10((x ** 2).sum() / ((x - x_hat) ** 2).sum().clamp(min=1e-30))
46
47print(f"per-tensor       SNR = {snr_db(A, A_recon_pt):6.1f} dB  scales: {scales_pt.numel():>6}")
48print(f"1x128 tile (act) SNR = {snr_db(A, A_recon_act):6.1f} dB  scales: {scales_act.numel():>6}")
49print(f"128x128 block    SNR = {snr_db(A, A_recon_w):6.1f} dB  scales: {scales_w.numel():>6}")

At Massive Scale: DeepSeek-V3's FP8 GEMM

DeepSeek-V3 trains a 671 B-parameter Mixture-of-Experts model on 14.8 T tokens, with most GEMMs executed in FP8. That works only because the team co-designed the FP8 format, the block layout, and a custom CUDA kernel that handles the scales correctly during the inner-product accumulation. The key facts:

Two design points are worth highlighting. First, the tile shape $(1, 128)$ for activations is chosen so the 128-element K-dimension of the GEMM's inner accumulate loop aligns with one tile-scale: the kernel loads 128 FP8 values, multiplies by one FP32 scale broadcast in a register, and accumulates in FP32. No re-scaling mid-loop, no scalar dependency chain inside the hot path. Second, the weight block $(128, 128)$ matches the canonical CUTLASS tile size on H100 for FP8 matmul, so each warp consumes exactly one weight scale per output tile.

What changes from the toy demo to a 671 B model

1. Per-tile scales become a tensor of their own. On a (16384, 4096) activation, the scale tensor is (16384, 32) FP32 — 2 MB. Across all activations in a transformer block, scale tensors sum to gigabytes. They are not free, and they have to be moved between memory hierarchies just like the FP8 payload.
2. Scale computation is on the critical path. amax must be computed before the FP8 write-back. On H100, fused-amax-and-quantise kernels (the "TE" primitive in NVIDIA Transformer Engine) write the FP8 tensor and the scale in one pass over the activation. A naive two-pass implementation doubles activation memory bandwidth and is unusable.
3. Backward needs E5M2, not E4M3. Activation gradients have a wider dynamic range than activations themselves — E5M2 trades mantissa for exponent and is the format DeepSeek uses on the backward pass. Each tile still gets its own scale; the only thing that changes is the underlying FP8 format. This split is invisible to the forward-pass code but critical for stability.
4. The amax history matters. Some FP8 systems (NVIDIA Transformer Engine) use a delayed scaling policy: compute amax during forward, store it, then use the previous step's amax for the current step's quantisation. This trades a small amount of accuracy (the amax can change by ±20% across steps) for the ability to fuse the FP8 write with the GEMM. DeepSeek-V3 uses just-in-time scaling — compute amax this step, use it this step — paying the bandwidth cost to avoid the accuracy hit.

Engineering Reality: Why 128 and Not 16 or 1024

The block size is a hyperparameter. Smaller blocks give better quantisation; larger blocks give better hardware throughput. Why does DeepSeek-V3 settle on exactly 128?

The choice is not arbitrary. 128 is the canonical K-dim of the H100's FP8 matrix-multiply-accumulate (MMA) instruction — each tensor-core instruction consumes a 16×16×16 fragment of FP8 operands, accumulated over the K dimension in steps of 16, and the kernel-level "K-block" that the warp loops over is 128. Aligning the quantisation block with the K-block means the scale broadcast is constant across the entire inner loop. Any smaller block forces a mid-loop scale change, which costs registers and breaks the dataflow. Any larger block lets one outlier's damage spread further than necessary.

Four engineering gotchas that the clean theory hides:

1. Tail padding. If $C$ is not a multiple of 128, the last tile is padded — and the padding zeros contribute to amax (they don't, since they are zero, but they do take up the scale's memory). Most training stacks ensure all dimensions are multiples of 128 by construction, which is why "hidden size 4096 / 8192 / 16384" is everywhere in modern LLM configs.
2. Mixed block shapes within one tensor. Some experimental schemes (e.g., NVIDIA's "current scaling") use 1×128 for activations and 1×K for the same tensor when it appears in a different GEMM. Each consumer of the tensor requires re-quantisation with its own preferred block shape. DeepSeek-V3 avoided this by standardising on (1, 128) for all activation appearances.
3. The activation-gradient bug. If you quantise activation gradients with a block shape that doesn't align with the gradient's upstream tensor's block shape, the backward GEMM produces subtly wrong values that only show up as slow training divergence. The fix is to keep block shapes consistent across forward and backward — the DeepSeek paper explicitly documents this constraint.
4. Communication-bound quantisation. For tensor-parallel and expert-parallel comms, the FP8 payload plus its scales travels the network. Sending the scales as BF16 instead of FP32 halves the metadata bandwidth; some systems do this for AllReduce while keeping FP32 scales for GEMM. Whether the savings are worth the extra accuracy loss depends on the network — H100 NVLink can ship FP32 scales for free, but slower interconnects (PCIe, Ethernet) benefit from BF16 metadata.

The deepest lesson. FP8 didn't fail in 2022 because the format was wrong — E4M3 is fine, the silicon is fast, the bandwidth savings are real. It failed because one global scale is the wrong abstraction for tensors whose magnitudes vary by orders of magnitude across positions. The DeepSeek-V3 contribution is not a new format or a new kernel; it is the observation that quantisation granularity is a design knob, and that the right setting — 1×128 for activations, 128×128 for weights — happens to align with the GEMM's natural tile structure on H100. The lesson generalises: every time a numerical method fails at scale, ask whether the failure is in the method or in the granularity at which the method is applied.