The Single-Token Prediction Bottleneck

Every transformer trained today does an enormous amount of work and is rewarded with a tiny amount of feedback. A 671-billion-parameter model spends roughly four trillion FLOPs to process a single token through its full forward and backward pass, and at the end of that journey it is graded on one question: what is the next token? One categorical target. A handful of bits of information per step. Most of the network's machinery is, in a precise information-theoretic sense, going to waste. This chapter is about why the wastage exists, what it costs, and how DeepSeek-V3 — along with a generation of speculative-decoding work — claws some of the lost signal back. This first section is just the diagnosis.

The bottleneck in one sentence. Next-token prediction makes the forward pass do the work of N layers and extracts the signal of one categorical guess. The compute scales with the model; the supervisory signal scales with the vocabulary. The gap between the two is the largest underused resource in large-model training.

The Asymmetry: Huge Forward Pass, Tiny Signal

Walk through what one training step actually does. A batch of sequences enters the model. For each of the $T$ positions in each sequence, every one of the $N$ transformer layers runs its full forward computation: self-attention, MLP, normalization, residual adds. Then the gradients flow back through the same N layers. The compute is proportional to $N \cdot T$, and it grows with the model.

At the very top of that stack, the final hidden state at position $t$ is projected to a vocabulary-sized logit vector and compared, by cross-entropy, against exactly one ground-truth token: the token at position $t+1$. The supervisory signal that this step delivers is bounded by $T \cdot \log_2(V)$ bits — and in practice it is much smaller, because natural text is nowhere near maximum entropy. The signal scales with the data, not the model.

Intuition: The 100-Step Exam Graded on One Number

Imagine a student sitting an exam that takes 100 steps of arithmetic to complete. The student does every step carefully. The teacher grades the work by reading only the final number — they never look at any intermediate result — and writes either a check or a cross at the bottom of the page. Over the course of a semester, the student improves on the final answer, but slowly, because each round of feedback is one bit of signal compressed from a hundred steps of effort. If the teacher had instead written a check or cross at every intermediate step, the student would have learned in ten exams what previously took a hundred.

A transformer is exactly this student. The forward pass computes a long chain of intermediate representations — early-layer hidden states contain low-level features, mid-layer hidden states contain syntactic structure, late-layer hidden states contain semantic content. By the time the final layer produces its logits, the model has done something analogous to a hundred steps of reasoning. The standard objective reads only the final logit vector and grades it on one cross-entropy target. Everything else is signal-free.

Why has the field tolerated this for so long? Two reasons. First, the single-target objective is mathematically clean — cross-entropy on the last hidden state has a tidy gradient and works out of the box. Second, for a long time the data side was the bottleneck: there was always more text to train on, so the question of "more signal per FLOP" was less urgent than the question of "more data, please." That changed once frontier labs started running out of high-quality tokens and once total training FLOPs began to dwarf data scaling. With both data and capital expensive, the bits-per-FLOP ratio became the new lever.

The Math: Supervisory Density per FLOP

Fix the model at $N$ total parameters and one training step that processes $T$ tokens. The standard estimate for FLOPs per step is:

$F_{\text{step}} = 6 \cdot N \cdot T$

The factor of 6 comes from $2$ FLOPs per parameter in the forward pass (one multiply plus one add, counted as two operations) and roughly $4$ FLOPs per parameter in the backward pass (one for the activation gradient, one for the weight gradient, each a multiply-add). Some analyses use a slightly different constant — 8 if you include the optimizer update — but 6 is the standard in scaling-law literature.

Now the supervisory side. Next-token prediction emits one cross-entropy target per position. The maximum information content of a categorical distribution over $V$ classes is $\log_2 V$ bits, achieved when the true distribution is uniform. Real text falls well below this ceiling, but as an upper bound:

$B_{\text{step}} = T \cdot \log_2 V$

Define the supervisory density $\rho$ as bits per FLOP:

$\rho = \frac{B_{\text{step}}}{F_{\text{step}}} = \frac{T \log_2 V}{6 N T} = \frac{\log_2 V}{6 N}$

The $T$'s cancel — a longer context does not change the density. Vocabulary buys only a logarithmic improvement. The only term that survives is $N$ in the denominator, which is exactly the observation we want: supervisory density falls linearly with model size. Big models are the worst possible thing for bits per FLOP.

The MTP lever, in one line

Suppose we attach $k$ output heads instead of one, so position $t$ predicts the tokens at $t+1, t+2, \dots, t+k$. The extra heads cost $O(k \cdot d \cdot V)$ FLOPs per step — small compared with the trunk's $O(N \cdot T)$ — so to a first approximation the FLOPs are unchanged. The bits scale linearly:

$\rho_k \approx \frac{k \cdot \log_2 V}{6 N}$

We multiplied the density by $k$ for free. Whether the model can use the extra k targets without confusing itself is the engineering question that the remaining sections of this chapter answer. Here we are only establishing the prize.

Manual Numerical Walkthrough

Numbers make the asymmetry vivid. Let us compute the supervisory density for three concrete configurations, by hand.

Visualizing the Bottleneck

The diagram below makes the compute-vs-signal asymmetry visceral. Each token column is a full forward + backward pass through $N$ layers — every amber cell lit, every step. The thin green lines at the bottom are the only supervisory signal that pass extracts. Hover any column to see what one position is asked to learn from one round of compute.

Three behaviors to try out. First, push the model-size slider up to 671B and watch the "bits per GFLOP" meter shrink — even though you have not changed the supervisory side at all, the compute side ballooned and the ratio collapsed. Second, push the predictions-per-position slider from 1 to 4. Compute barely budges; the supervisory bar grows in lockstep with k. The density meter climbs. Third, try the preset buttons: "DeepSeek-V3 / NTP" gives the tiny ratio that motivated MTP, and "DeepSeek-V3 / MTP-2" shows the resulting density without redrawing a single layer of the trunk.

Plain Python: Counting the Signal

Before we touch a real loss function, it is worth writing the bookkeeping directly. The code below does no training — it simply counts, in NumPy, what the standard NTP step costs and what it produces. The same formulas scale, unchanged, from the toy case to DeepSeek-V3.

1import numpy as np
2
3# A toy transformer's training step. We do NOT run the network here — we just
4# count the FLOPs it would do and the supervisory bits we extract from one step.
5
6# Model + data dimensions.
7N_PARAMS  = 7e9          # 7B parameters
8T         = 4096         # context length (tokens per sequence)
9V         = 50_000       # vocabulary size
10BATCH     = 1            # per-GPU micro-batch for the demo
11
12# (a) FLOPs per training step. Standard estimate:
13#     forward + backward ≈ 6 * N_params per processed token.
14flops_per_step = 6 * N_PARAMS * T * BATCH
15
16# (b) Supervisory signal. Standard next-token prediction supplies exactly
17#     one cross-entropy target per position. Upper bound per target is log2(V)
18#     bits (the maximum entropy a categorical with V outcomes can carry).
19bits_per_step = T * BATCH * np.log2(V)
20
21# (c) Supervisory density: how many bits of signal per gigaflop of compute.
22density = bits_per_step / (flops_per_step / 1e9)
23
24print(f"FLOPs/step      = {flops_per_step:.3e}")
25print(f"Bits/step (max) = {bits_per_step:.3e}")
26print(f"Density         = {density:.6f} bits / GFLOP")
27
28# (d) What if we predicted k targets per position instead of 1?
29#     The forward pass cost is unchanged (one extra LM head ≪ a 70-layer trunk).
30#     The supervisory bits scale linearly with k.
31for k in [1, 2, 3, 4]:
32    bits_k    = k * T * BATCH * np.log2(V)
33    density_k = bits_k / (flops_per_step / 1e9)
34    print(f"k = {k}: density = {density_k:.6f} bits / GFLOP")

Two structural points are worth marking. First, the loop over k is the cheapest possible preview of MTP: nothing about the network changes; we simply multiply the bits term by k and re-divide. The fact that this trivial edit produces a multiplicatively-better ratio is the entire reason MTP is a serious research direction. Second, the FLOPs estimate uses the canonical $6 \cdot N \cdot T$ rule because that is what scaling-law papers use; if you have read Hoffmann et al.'s Chinchilla paper, this is exactly their accounting.

Sanity check. If you set N_PARAMS = 1 and V = 2, the bits/step becomes $T \cdot 1 = T$ and the FLOPs/step becomes $6T$. Density = 1/6 bits/FLOP — one bit per six operations. That is the theoretical best a transformer could ever get at this objective. Any real model lands many orders of magnitude below it, and the gap is the cost of using the transformer as a function approximator.

PyTorch: The Standard NTP Loss in One Line

Now the same idea, instantiated in PyTorch. This is the loss every modern LLM is trained with, give or take a label-smoothing parameter. Notice how short it is — that brevity is exactly what we are about to enrich.

1import torch
2import torch.nn.functional as F
3
4def ntp_loss(
5    logits: torch.Tensor,  # (B, T, V) — model output for every position
6    inputs: torch.Tensor,  # (B, T)    — token IDs
7) -> torch.Tensor:
8    """
9    Standard next-token prediction loss. The hidden state at position t is
10    asked to predict the token at position t+1. The last position has no
11    target, so we shift logits and inputs and align them.
12    """
13    # (a) Drop the last position of logits — it has no successor to predict.
14    shift_logits = logits[:, :-1, :]              # (B, T-1, V)
15    # (b) Drop the first position of inputs — it has no predecessor to be predicted.
16    shift_labels = inputs[:, 1:]                  # (B, T-1)
17
18    # (c) Cross-entropy over the full vocabulary, averaged across all
19    #     (B * (T-1)) positions. Each averaged term is ONE supervisory target.
20    loss = F.cross_entropy(
21        shift_logits.reshape(-1, shift_logits.size(-1)),
22        shift_labels.reshape(-1),
23    )
24    return loss
25
26# Sanity check on the toy: 2 sequences, 8 tokens, vocab of 100.
27torch.manual_seed(0)
28B, T, V = 2, 8, 100
29logits  = torch.randn(B, T, V)
30inputs  = torch.randint(0, V, (B, T))
31
32loss = ntp_loss(logits, inputs)
33# How many supervisory targets contributed to this scalar?
34n_targets = B * (T - 1)
35print(f"loss = {loss.item():.4f}")
36print(f"supervisory targets reduced into this scalar = {n_targets}")
37print(f"bits/target upper bound = {torch.log2(torch.tensor(float(V))).item():.3f}")

Two subtleties worth marking, both about how this loss interacts with the rest of the training step:

1. The reduction is a mean, not a sum. Each of the $B(T-1)$ positions contributes $1/(B(T-1))$ of the final scalar. This means the loss magnitude does not grow with context length, which is convenient — but it also means the gradient delivered per token is a tiny fraction of the total loss. The asymmetry we counted at the FLOP level is faithfully preserved at the gradient level: more positions means more shared weight on each backward signal.
2. The shift is non-negotiable. Logits at position t predict the token at t+1. If you forget to shift, you train the model to predict itself — trivial accuracy, zero useful gradient. This is a common bug in hand-rolled training loops and the reason most frameworks ship a wrapped loss with the shift baked in.

What Changes at Massive Scale

At toy scale the bottleneck is a curiosity. At frontier scale it is a defining constraint of the training economy. Three things change as N grows from millions to hundreds of billions of parameters.

Compute becomes the dominant cost

At 100M parameters, a training run might cost a few thousand GPU-hours and the data pipeline is the bottleneck — you tune tokenization, dedup, and domain mixing because that is where the gains are. At 671B parameters, a single training run costs tens of millions of GPU-hours, and the data pipeline is solved enough that every additional bit of supervisory signal per FLOP translates directly into millions of dollars of compute equivalent. The density ratio stops being academic.

The model can absorb more signal than it gets

Empirically, 100B+ models trained on NTP show clear evidence of under-supervision: probes of intermediate layers reveal predictive information about tokens many positions ahead, but the model is never asked to use it. Multi-token prediction does not teach the model new tricks — it unlocks tricks the model has already learned in service of a one-step objective. The capacity is already there; only the supervisory signal is missing.

Inference cost lines up with training-time choices

A model trained to predict $k$ tokens ahead at every position is, almost by accident, also a model whose internal representations are well-suited for speculative decoding at inference — draft k tokens in parallel and accept the prefix that matches a verifier. Section 7.5 makes this connection explicit. What looks like a training-time density argument in this section turns into a 2–4× inference speedup later, with no extra parameters and no separate draft model required.

Engineering Reality and the Door MTP Opens

Three practical observations sit on top of the math, and each one will recur through the rest of this chapter.

1. The bottleneck is not a bug; it is a default. F.cross_entropy on shifted logits is so easy and so robust that it became the universal recipe. There is nothing "wrong" with it — every model in production uses it. The point is that the default leaves information on the table, and the cost of leaving that information on the table grows with model scale. MTP is not fixing a bug; it is collecting on a debt.
2. Naïve parallel k-target prediction does not work. The obvious move is to attach k independent output heads, each predicting $t+i$ from the same hidden state. This is what Section 7.2 examines, and it fails: heads compete for the same representation, gradients clash, and the model often does worse than NTP. DeepSeek-V3's sequential causal MTP from Section 7.3 is the construction that actually works, and the rest of the chapter is about why.
3. The right loss coefficient is small. Even when the architecture is right, the MTP heads are auxiliary — the main objective is still next-token prediction. DeepSeek-V3 weights MTP losses at roughly $\lambda = 0.3$ early in training and decays to $0.1$ by the end. Section 7.4 derives why this schedule matters, and how the wrong choice silently destroys the gains from the architecture in Section 7.3.

One sentence to carry forward into the rest of the chapter: the single-token objective is a compute-efficient learner of a signal-inefficient problem; multi-token prediction is the cheapest known way to fix the supervisory side without touching the trunk. Everything that follows in Chapter 7 is the careful engineering of that fix.