MoE Scaling Laws

Section 9.1 derived the Chinchilla scaling laws for dense transformers — the famous result that, for a given compute budget, the loss-minimising allocation is roughly twenty tokens per parameter. Those laws are a bedrock prediction tool, and every frontier lab uses them to size dense models. But Chinchilla assumes that every parameter touches every token. Mixture- of-Experts models violate that assumption deliberately. In an MoE, most of the parameters sit idle on any given token — they only wake up when the router sends a token their way. The cost-per-token is decoupled from the capacity-per-token, and the entire Chinchilla framework needs an extra dimension to describe it.

The thesis of this section. Dense scaling laws have two knobs: parameters $N$ and tokens $D$. MoE scaling laws have three: activated parameters $N_{\text{act}}$, tokens $D$, and granularity $G = N_{\text{total}} / N_{\text{act}}$. The third knob is what lets DeepSeek-V3 ship 671 B parameters for the per-token cost of a 37 B dense model — and the scaling law tells us exactly how much that knob is worth.

The Real Problem: Chinchilla Breaks at G > 1

Run a Chinchilla optimiser on a 671 B-parameter target and it will prescribe roughly $20 \cdot 671 \approx 13.4$ T tokens, costing on the order of $6 \cdot 671 \cdot 10^9 \cdot 13.4 \cdot 10^{12} \approx 5.4 \cdot 10^{25}$ FLOPs. That is more compute than any single lab spent in 2024. Yet DeepSeek-V3 trained a 671 B model on 14.8 T tokens at roughly $3 \cdot 10^{24}$ FLOPs — almost twenty times cheaper than Chinchilla predicts. The dense law does not just mispredict the cost; it mispredicts which side of the budget constraint the model can live on. Something is missing from the formula.

What is missing is the distinction between two things Chinchilla treats as one:

Chinchilla's scaling law uses one $N$ for both the cost column and the capacity column. MoE has to split them. Empirical work by Clark et al. (2022), Krajewski et al. (2024), and the DeepSeek team has converged on a clean three-parameter extension: keep Chinchilla's functional form, replace the capacity term with effective capacity, and let the data decide how much of the extra MoE parameters actually contribute.

Intuition: Two Knobs Instead of One

Think of a dense transformer as a single big specialist. Every token consults the same expert. Its cost-per-token is the specialist's size, and so is its capacity. You cannot grow capacity without growing cost.

An MoE is a panel of small specialists with a triage doctor. Each token sees the triage doctor (the router), gets routed to $k$ specialists out of $E$, and only those $k$ specialists do work. Cost-per-token is the size of the $k$ firing specialists. Capacity is the size of the whole panel. The two are now different numbers — that is the entire design freedom.

Granularity $G$ is the panel-to-firing-doctor ratio. $G = 1$ is a single doctor — a dense model. $G = 18$ is the DeepSeek design: 256 experts, top-8 routing, so on every token $8 / 256 \approx 1/32$ of the parameters fire, modulated by the shared-expert design that brings that ratio up to roughly $1 / 18$ in effective FLOPs.

The Mathematical Form of an MoE Scaling Law

Start with Chinchilla. Hoffmann et al. (2022) fit:

$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$

with $E \approx 1.69$ (the irreducible text entropy), $\alpha \approx 0.34$, $\beta \approx 0.28$. The first term is unreachable language entropy; the second falls as the model gets bigger; the third falls as the training set grows.

Clark et al. (2022) — and more precisely Krajewski et al. (2024) — showed that the MoE form is the same equation with the capacity term replaced by an effective parameter count:

$L(N_{\text{act}}, D, G) = E + \frac{A}{(N_{\text{act}} \cdot G^{\gamma})^{\alpha}} + \frac{B}{D^{\beta}}$

with $G = N_{\text{total}} / N_{\text{act}}$ the granularity and $\gamma \in (0, 1)$ the MoE conversion exponent. Set $G = 1$ and you recover dense Chinchilla exactly. Set $G > 1$ and you trade total parameters for activated cost at a discount governed by $\gamma$.

The effective parameter count

Define $N_{\text{eff}} = N_{\text{act}} \cdot G^{\gamma}$. For DeepSeek-V3 ( $N_{\text{act}} = 37$ B, $G \approx 18.1$):

$N_{\text{eff}} = 37 \cdot 10^{9} \cdot 18.1^{0.35} \approx 1.05 \cdot 10^{11}$

Read: a 37 B-activated MoE with granularity 18 behaves, in the scaling-law fit, like a dense 105 B-effective-parameter model. It is not as good as a true dense 671 B (which would be $N_{\text{eff}} = 671$ B), because $\gamma < 1$. But it costs the FLOPs of a 37 B model, and it punches at the loss of a 105 B model. That is the MoE bargain in one equation.

The compute-optimal point with G held fixed

Training compute is paid only on activated experts:

$C = 6 \cdot N_{\text{act}} \cdot D$

Substitute $D = C / (6 N_{\text{act}})$ into the loss formula and minimise over $N_{\text{act}}$ at fixed $G$ and $C$. The Chinchilla algebra goes through unchanged. The optimum sits where the capacity and data partial derivatives balance:

$N_{\text{act}}^{\star} \propto C^{\beta / (\alpha + \beta)} \cdot G^{-\gamma \beta / (\alpha + \beta)}$

$D^{\star} \propto C^{\alpha / (\alpha + \beta)} \cdot G^{\gamma \beta / (\alpha + \beta)}$

Two facts to read off. First, at fixed compute, increasing $G$ shifts the optimum toward smaller $N_{\text{act}}$ and more tokens. That is exactly the DeepSeek design — a relatively modest 37 B activated, trained on 14.8 T tokens. Second, with $\gamma = 0.35$ and $\alpha = 0.34, \beta = 0.28$, the scaling exponent of $G$ in the optimal $N_{\text{act}}$ is roughly $-0.16$ — small, but enough to push from a dense 70 B compute-optimum down to a 37 B-activated MoE optimum at the same FLOPs.

Manual Numerical Walkthrough

Three configurations, same compute budget, all derived by hand.

Visualizing the MoE Frontier

The widget below lets you set activated parameters $N_{\text{act}}$, granularity $G$, and a compute budget $C$. It computes the predicted MoE loss at the compute-optimal $D$, and overlays the dense Chinchilla curve (i.e. $G = 1$ with the same activated count) for comparison. The presets reproduce the three numerical configurations above.

Three movements to lock in. First, pull $G$ down to 1 — the two curves merge, because $G = 1$ recovers dense Chinchilla exactly. Second, push $G$ up to 18 with the DeepSeek preset — the MoE curve drops uniformly below the dense curve across every compute budget you slide through. Third, drop $N_{\text{act}}$ to 7 B with G = 16 (Switch preset) — the MoE curve rises above the dense curve at small compute, because the capacity term dominates when $N_{\text{act}}$ is too small to back up the granularity gain.

Plain Python: Predicting Loss for a Candidate MoE

The plain-Python implementation evaluates the MoE scaling law for three real configurations and picks the compute-optimal token count for each. No PyTorch — just math.log and a for-loop over candidate $D$ values. This is exactly the code a scaling-law team runs before they decide what model to spend a $100M training run on.

1import math
2
3# Fitted MoE scaling-law constants (Krajewski et al. 2024, illustrative).
4E      = 1.69     # irreducible loss
5A, B   = 406.4, 410.7
6ALPHA  = 0.34     # capacity exponent
7BETA   = 0.28     # data exponent
8GAMMA  = 0.35     # how much MoE 'total params' convert to effective capacity
9
10def predicted_loss(n_act_B, d_B, G):
11    """
12    n_act_B : activated params per token, in billions
13    d_B     : training tokens, in billions
14    G       : granularity factor = N_total / N_act
15    """
16    N_eff = (n_act_B * 1e9) * (G ** GAMMA)   # effective capacity
17    D     = d_B * 1e9
18    return E + A * N_eff**(-ALPHA) + B * D**(-BETA)
19
20def compute_flops(n_act_B, d_B):
21    # Training compute is dominated by the activated experts only.
22    return 6 * (n_act_B * 1e9) * (d_B * 1e9)
23
24def compute_optimal(n_act_B, G, C_budget):
25    """Sweep D in log space; return (D*, L*) under the given budget."""
26    best = (None, math.inf)
27    for k in range(0, 251):
28        d_B = 10 ** (k / 50.0)                 # 1B .. 1e5 B = 100T
29        if compute_flops(n_act_B, d_B) > C_budget:
30            break
31        L = predicted_loss(n_act_B, d_B, G)
32        if L < best[1]:
33            best = (d_B, L)
34    return best
35
36# Three configurations at the same training compute (3.4e24 FLOPs)
37C = 3.4e24
38
39dense_70B            = compute_optimal(70, G=1,    C_budget=C)
40deepseek_37B_x_G18   = compute_optimal(37, G=18.1, C_budget=C)
41switch_7B_x_G16      = compute_optimal( 7, G=16,   C_budget=C)
42
43for name, (d, L) in [
44    ("Dense 70B           ", dense_70B),
45    ("DeepSeek 671B / 37B ", deepseek_37B_x_G18),
46    ("Switch 112B / 7B    ", switch_7B_x_G16),
47]:
48    print(f"{name}  D* = {d:6.0f} B tokens   L* = {L:.3f}")

Two structural details. First, the FLOPs formula$C = 6 N_{\text{act}} D$ is the entire reason MoE is economically viable. If you accidentally use $6 N_{\text{total}}$ in your budget accounting, every MoE looks ruinous and you stop building them. Second, the constants $\alpha, \beta, \gamma$ are not universal — they need to be re-fit on small-scale runs of your architecture, your data, and your tokenizer. The numbers above are illustrative; in production you would run a sweep of ~20 small proxy models, fit the three exponents by non-linear least squares, then transfer to the full run.

PyTorch: Searching the (N_act, G) Frontier

For a real frontier search you do not evaluate three configurations — you evaluate a grid of hundreds, then a finer grid in the neighbourhood of the optimum. PyTorch's broadcasting makes the whole grid a single vectorised call, and torch.argmin pulls the optimum out in microseconds.

1import torch
2
3# Same scaling-law constants as before, now on tensors so we can sweep
4# the entire (N_act, G) frontier in a single batched evaluation.
5E      = 1.69
6A, B   = 406.4, 410.7
7ALPHA, BETA, GAMMA = 0.34, 0.28, 0.35
8
9C_budget = 3.4e24  # training FLOPs ~ DeepSeek-V3
10
11def loss_grid(n_act_B, G, D_B):
12    """All three inputs are tensors of compatible shape."""
13    N_eff = (n_act_B * 1e9) * G.pow(GAMMA)
14    D     = D_B * 1e9
15    return E + A * N_eff.pow(-ALPHA) + B * D.pow(-BETA)
16
17# 1) Grid over candidate MoE configurations.
18n_act = torch.tensor([7., 13., 22., 37., 70., 120.])      # activated B
19g     = torch.tensor([1., 2., 4., 8., 16., 24., 32.])     # granularities
20
21# 2) For each (N_act, G) cell, derive the maximum D affordable.
22#    FLOPs = 6 * N_act * D  =>  D_max = C / (6 * N_act).
23D_max = C_budget / (6 * n_act[:, None] * 1e9) / 1e9       # in B tokens
24# Broadcast g over rows so D_max has shape (|n_act|, |g|).
25D_max = D_max.expand(-1, g.numel())
26
27# 3) Loss at the compute-optimal D (= D_max under this constraint).
28N_act_B  = n_act[:, None].expand(-1, g.numel())
29G_grid   = g[None, :].expand(n_act.numel(), -1)
30L_grid   = loss_grid(N_act_B, G_grid, D_max)
31
32# 4) Find the global minimum over the (N_act, G) frontier.
33flat = L_grid.flatten()
34k    = torch.argmin(flat)
35i, j = divmod(int(k), g.numel())
36print(f"Compute-optimal MoE @ {C_budget:.1e} FLOPs:")
37print(f"  N_act = {n_act[i].item():6.1f} B")
38print(f"  G     = {g[j].item():6.1f}x  -> N_total = {(n_act[i]*g[j]).item():6.1f} B")
39print(f"  D     = {D_max[i, j].item():6.0f} B tokens")
40print(f"  L*    = {L_grid[i, j].item():.3f}")

Three observations worth marking, all about how a real lab uses this loop:

1. The grid is the search space, not the search algorithm. Brute-force evaluation of a 50 × 50 grid is ten thousand loss evaluations — milliseconds on a CPU. There is no need for gradient-based optimisation here; the loss surface is smooth and low-dimensional, and a dense grid finds the optimum more reliably than any clever optimiser.
2. Always include the dense column. A frontier search that does not include $G = 1$ in the grid cannot tell you whether the MoE is actually winning. The dense column is the falsifier; every cell of the MoE grid is good news only relative to that anchor.
3. Re-fit the exponents at scale before launch. The grid's optimum depends on three numbers ($\alpha, \beta, \gamma$) fit on small proxies. Those numbers drift with scale — γ in particular tends to drop slightly as activated count grows, because expert overlap increases. DeepSeek's reported procedure is to fit on 280 M – 1 B proxies, then re-fit a single correction term from one or two 16 B-activated runs before committing to the 37 B-activated production model.

At Massive Scale: How DeepSeek Picked 671B / 37B

Put the scaling law against the real DeepSeek-V3 numbers. The public reports give us: $N_{\text{total}} = 671$ B, $N_{\text{act}} \approx 37$ B, $D = 14.8$ T tokens, training compute on the order of $3.0 \cdot 10^{24}$ FLOPs. Plug those into the law and you get:

$G = 671 / 37 \approx 18.1$

$N_{\text{eff}} = 37 \cdot 10^{9} \cdot 18.1^{0.35} \approx 1.05 \cdot 10^{11}$

$L \approx 1.69 + 406.4 \cdot (1.05 \cdot 10^{11})^{-0.34} + 410.7 \cdot (1.48 \cdot 10^{13})^{-0.28} \approx 1.96$

Now ask: what would a dense model at the same compute budget have delivered? Solving Chinchilla for $C = 3.0 \cdot 10^{24}$ gives a compute-optimal dense $N \approx 65$ B, $D \approx 7.7$ T tokens, $L \approx 2.02$. The MoE wins by $\sim 0.06$ nats — a meaningful margin that compounds over downstream evaluations into the kind of capability gap that wins benchmarks.

What changes at the frontier

Read the table sideways: every MoE bottleneck that exists is on the memory and communication rows. The compute-and-latency rows look just like dense — except smaller. That is why MoE is an engineering win at frontier scale and a nuisance at small scale: small models do not have a memory bottleneck to relieve.

Engineering Reality and Gotchas

MoE scaling-law fits are clean equations, but three production failure modes earn their flags:

1. γ is not constant across architectures. The conversion exponent $\gamma$ depends on how independent the experts actually are. Switch (one-expert routing) sits around $\gamma \approx 0.25$; DeepSeek-style fine-grained experts with top-8 routing push it toward $0.38$. Use the wrong $\gamma$ and your prediction of the 671 B model's loss can be off by 0.1 nats — enough to abort a $50 M run that would in fact have succeeded.
2. Capacity factor and load balancing affect the effective N_act. The clean formula assumes every token gets exactly its k routed experts. In practice, capacity overflow drops some tokens and load imbalance burns experts. Both push the effective $N_{\text{act}}$ below its nominal value. The auxiliary-loss-free balancing of chapter 6 was designed in part so that scaling-law predictions would actually hold at frontier scale.
3. The data exponent β depends on quality, not just quantity. The fit above uses a single $\beta$, but real $\beta$ shifts when the mixture changes (chapter 8.4). Increasing the math weight raises $\beta_{\text{math}}$ at the cost of $\beta_{\text{web}}$. Production teams carry a separate $\beta_i$ per domain and aggregate; treating β as a single number is a 0.02-nat mispredict, small but enough to flip a Pareto comparison.

The one sentence to carry forward: MoE scaling laws turn one Chinchilla knob into three — $N_{\text{act}}$, $D$, and $G$ — and the third one is what lets a 671 B model train for the cost of a 37 B model and ship for the latency of a 37 B model — which is the entire reason every frontier release since 2024 has been an MoE.