Expert Parallelism and Cross-Node All-to-All

We have now seen four parallelism strategies — data, tensor, pipeline, and DualPipe's overlap of compute and communication — and built each one on top of the same primitive: a collective that synchronises tensors across ranks. Expert parallelism is the fifth, and it is the one that cracks open the path to 671B parameters. It is also the one that turns the cluster's network — not its compute — into the bottleneck. This section is the story of that bottleneck and DeepSeek's deceptively small fix.

The thesis. Expert parallelism makes each MoE layer shoot two all-to-all collectives over the network per forward pass. In a multi-node cluster the link that limits training throughput is no longer NVLink — it is the InfiniBand fabric between nodes, which is roughly an order of magnitude slower. Node-limited routing, a one-line change to the router, caps the InfiniBand traffic to a constant multiple of the NVLink traffic so the two finish at the same time. That single equation is what makes DeepSeek-V3's pre-training feasible on commodity 8-GPU nodes.

Where Expert Parallelism Sits in the Stack

Before zooming into the network, let us place expert parallelism among the strategies you already know. The four core parallelism axes partition four different things across the cluster:

Tensor parallelism is deliberately kept on-node because its all-reduce sits in the critical path of every matmul — losing a factor of ten on bandwidth would be catastrophic. Pipeline parallelism crosses nodes cheaply because the messages are small (activations, not weights). Data parallelism crosses nodes for the gradient all-reduce, but modern sharded variants (ZeRO-3, FSDP) hide most of that behind compute. Expert parallelism is the new entry, and it is the only collective in the stack that must traverse the cross-node fabric — there is no on-node shortcut, because the chosen expert can live anywhere.

The Real Problem: Two Networks, One Pinch Point

Inside an H800 node, eight GPUs share an NVLink/NVSwitch fabric that delivers around 600 GB/s of effective bisection bandwidth per GPU. Between nodes the cluster uses InfiniBand — usually one or two 200–400 Gbps HCAs per node — which works out to about 50 GB/s per GPU once you amortise NIC sharing. The two numbers are not close. NVLink is roughly $12\times$ faster than IB. Any algorithm that pushes the same number of bytes across both will spend twelve times longer on the IB hop than the NVLink one. That is the whole story.

Now consider a naive EP layout. We have $N$ nodes, each with 8 GPUs, holding $E$ routed experts spread evenly. A token activates top-$k$ experts. Assuming uniform routing, the probability that any one chosen expert sits on the token's home node is $1/N$. So the probability it sits off the home node is $1 - 1/N$. For $N = 8$, that is 87.5%. Almost every chosen expert is across the network.

The damage compounds because top-k routing fires $k$ of these per token. Each cross-node hop drags a row of the activation tensor — typically $d = 7168$ values at 2 bytes each, so 14 KB — across IB. With a per-device micro-batch of $T = 4096$ tokens at $k = 8$ and $N = 8$, that is $4096 \cdot 8 \cdot 0.875 \cdot 14\,\text{KB} \approx 400\,\text{MB}$ of IB traffic per device per dispatch — and there is a matching combine on the way back, doubling it. At 50 GB/s that is 16 ms of pure network time per MoE layer. Multiply by 58 MoE layers and the forward pass alone wants ≈ 0.9 s of IB time. The GPUs are idle the whole while.

Intuition: A City With Both Subways and Airplanes

Forget GPUs for a moment. Imagine a courier service operating in eight cities. Within a city there is a subway: cheap, fast, near-infinite capacity. Between cities the courier must fly: expensive, slow, bandwidth-limited. A package's destination is randomly chosen among the eight cities. If we let every package choose freely, 7 out of 8 packages get on a plane. The airport is the bottleneck — the planes always have a queue, the subways are mostly empty.

Now suppose, before the package leaves the warehouse, we attach a rule: a single package can visit at most four cities on its journey. If you need to drop off four parcels, you must choose four cities, but never a fifth. Inside those four cities you can take as many subway rides as you like. What changes? Each package now boards at most one plane (to the chosen second city) plus the subway hops within each visited city. Aggregate plane traffic falls by a factor of two — no, by a factor of $(N/M)$ where $M = 4$. The subways stay just as fast. The airport stops being the bottleneck.

That cap is node-limited routing. The cities are nodes; the airport is the InfiniBand NIC; the subway is NVLink; the packages are activation rows. Capping the number of nodes a token may touch is the only knob in the system that operates on the bandwidth-expensive resource without sacrificing expressive top-k routing.

The Math: All-to-All Cost and the Cross-Node Ratio

Let us model the cost of one MoE dispatch on a cluster of $N$ nodes, each with $G$ GPUs. Let $T$ be the number of tokens held by one GPU before dispatch, $d$ the activation hidden dimension, $k$ the top-k count, and $b$ the activation element size in bytes (typically $b = 2$ for BF16).

For unconstrained routing each of the $k$ chosen experts is equally likely to live on any node, so the expected number of distinct nodes a token must reach is

$\mathbb{E}[\text{distinct nodes}\;|\;\text{unconstrained}] = N \cdot \left(1 - \left(1 - \tfrac{1}{N}\right)^{k}\right).$

For $N = 8$ and $k = 8$ this evaluates to about $5.25$ nodes per token, of which exactly one is the home node. So a typical token travels to $5.25 - 1 = 4.25$ remote nodes — and a copy of its activation row must cross IB once per remote node.

Multiplying through, the number of activation rows leaving one GPU over InfiniBand in a single dispatch is

$R_{\text{IB}} = T \cdot \left[N \cdot \left(1 - \left(1 - \tfrac{1}{N}\right)^{k}\right) - 1\right],$

and the IB byte budget per dispatch is $B_{\text{IB}} = R_{\text{IB}} \cdot d \cdot b$. The combine on the way back is symmetric, so the round-trip cost is$2 B_{\text{IB}}$.

On the NVLink side, the analogous count of intra-node per-token transfers depends on how many of the $k$ picks landed inside the home node (or, with the hierarchical algorithm we will build, inside any node the token already visits). The cost model implemented in the cost-model simulator below is a tractable closed form; the key proportion is the ratio

$\rho \;\equiv\; \frac{B_{\text{IB}}}{B_{\text{NVLink}}}.$

We want $\rho$ to be at most the link speed ratio $\beta_{\text{NVLink}} / \beta_{\text{IB}} \approx 12$ — i.e. we want IB to carry at most as much traffic per second as NVLink, so that wall time on the two links matches and the dispatch is bandwidth-balanced.

DeepSeek's Trick: Node-Limited Routing

DeepSeek-V3 introduces a hard structural constraint on the router: a token may activate experts on at most $M$ distinct nodes. In V3 they fix $M = 4$ with $N = 8$. The implementation is a single masking step that runs before the top-k selection:

• For each token, score every expert with the router as usual: $s_e = (\text{softmax of } x^\top W_r)_e$.
• For each node $n$, compute the top-2 expert score on that node and call its sum $\mathrm{aff}_n$ — the node's "affinity" for this token.
• Keep the top $M$ nodes by $\mathrm{aff}_n$; zero out all expert scores on the remaining $N - M$ nodes.
• Run top-$k$ selection over the surviving scores. The token now visits at most $M$ nodes by construction.

The router is still differentiable — masking is just multiplying by a gradient-free indicator — and the bias-term load balancer from §6 keeps the per-node affinities from collapsing onto a few favourite nodes. Quality loss in V3 ablations is below the measurement noise on downstream evals; the bandwidth savings are not.

With the cap in place, the expected distinct-node count for one token is at most $M$, so

$R_{\text{IB}}^{\text{node-limited}} \le T \cdot (M - 1).$

Crucially, this bound is independent of $N$: doubling the cluster from 8 to 16 nodes does not increase per-device IB traffic so long as $M$ stays fixed. The collective scales with the cap, not with the cluster.

This is what "near-perfect computation-communication overlap" means. Pick the smallest $M$ such that IB wall time matches the 10 ms or so of expert compute per layer. For V3-scale workloads, that $M$ turns out to be 4. The cost-model simulator in the next section makes this concrete.

Manual Numerical Walkthrough

Set $N = 8$, $k = 8$, $T = 4096$, $d = 7168$, $b = 2$ (BF16). We compare two routings.

Case A: unconstrained top-8 across N = 8 nodes

Expected distinct nodes per token:$8 \cdot (1 - (7/8)^8) = 8 \cdot 0.656 = 5.25$.

Cross-node hops per token: $5.25 - 1 = 4.25$.

Per-device rows over IB per dispatch: $4096 \cdot 4.25 = 17{,}408$.

Per-device bytes over IB per dispatch: $17{,}408 \cdot 7168 \cdot 2 \approx 250\,\text{MB}$. Round trip: $500\,\text{MB}$.

Wall time at 50 GB/s: $500\,\text{MB} / 50\,\text{GB/s} \approx 10\,\text{ms}$. Per MoE layer alone. Across 58 layers, the forward pass alone needs ~580 ms of pure IB time. A whole training step would be IB-bound.

Case B: node-limited with M = 4

Distinct nodes per token capped at $M = 4$. Cross-node hops per token: $4 - 1 = 3$.

Per-device rows over IB per dispatch: $4096 \cdot 3 = 12{,}288$.

Per-device bytes over IB per dispatch: $12{,}288 \cdot 7168 \cdot 2 \approx 176\,\text{MB}$. Round trip: $352\,\text{MB}$.

Wall time at 50 GB/s: $352\,\text{MB} / 50\,\text{GB/s} \approx 7\,\text{ms}$ per MoE layer.

That alone is a 30% reduction in IB time without any change to expert count or model dimension. Push $M$ further down to 2 and IB drops to ~3.5 ms — but quality starts to degrade because each token has only two candidate node-pools to pick experts from. DeepSeek's ablations land on $M = 4$ as the sweet spot.

Visualizing Dispatch and the Bandwidth Budget

First, the mechanism. The animation below replays the five stages of an EP dispatch — local, route, dispatch all-to-all, expert compute, combine all-to-all — on a four-GPU cluster. Watch the colored tokens leave their home GPU during Dispatch, hit the expert that owns them during Compute, then travel back on Combine. The same logic scales to a multi-node cluster: just imagine each GPU column belonging to one of two nodes, with the all-to-all crossing the node boundary.

Now the cost. The simulator below puts the math from The Math behind a set of sliders. Drag $M$ down from $N$ and watch the IB bar shrink while the NVLink bar barely moves — that is exactly the asymmetry that node-limited routing exploits. The "Find smallest M that fits IB" button picks the smallest cap that keeps IB under the 10 ms compute budget, the same logic DeepSeek's team used to land on $M = 4$.

Two patterns to take away. First, IB saturation grows almost linearly with $(N - 1)/M$: small clusters with no cap and large clusters with a cap of 4 sit at similar IB load. Second, NVLink barely moves as you change $M$ because the intra-node bundle has to be moved either way — node-limited routing is a pure win on the slow link.

Plain Python: A Two-Phase All-to-All

Below is the algorithm as plain NumPy — no torch.distributed yet. Eight simulated ranks across two nodes, top-2 routing over sixteen experts. The two phases are visible as separate loops: first we bundle by destination node (the IB hop), then we re-sort bundles to the exact destination GPU within each node (the NVLink hop).

(4, 4)

(16, 4)

(T_local, k) = (4, 2)

1import numpy as np
2
3# Cluster: 2 nodes, 4 GPUs/node = 8 ranks. Experts: 16 (2 per GPU).
4NODES, GPUS_PER_NODE = 2, 4
5WORLD = NODES * GPUS_PER_NODE          # 8
6E, k = 16, 2                            # 16 routed experts, top-k = 2
7experts_per_gpu = E // WORLD
8node_of = lambda r: r // GPUS_PER_NODE
9owner = lambda e: e // experts_per_gpu  # gpu id (0..7) that owns expert e
10
11# Per-GPU local batch: 4 tokens of dim d=4.
12d, T_local = 4, 4
13rng = np.random.default_rng(0)
14local_x = [rng.standard_normal((T_local, d)) for _ in range(WORLD)]
15
16# Replicated router (each rank runs it on its own tokens).
17W_router = rng.standard_normal((E, d)) * 0.1
18
19def route_topk(x, k):
20    logits = x @ W_router.T            # (T, E)
21    return np.argsort(-logits, axis=1)[:, :k]   # (T, k)
22
23# ---- PHASE 1: build per-destination-NODE bundles (IB hop) ----
24ib_send_buffers = [[None] * NODES for _ in range(WORLD)]
25for r in range(WORLD):
26    picks = route_topk(local_x[r], k)        # (T, k)
27    for dest_node in range(NODES):
28        # Tokens with at least one expert living on dest_node.
29        mask = np.array([
30            any(node_of(owner(e)) == dest_node for e in picks[t])
31            for t in range(T_local)
32        ])
33        ib_send_buffers[r][dest_node] = local_x[r][mask]
34
35# IB all-to-all: every rank ships ONE bundle per remote node.
36ib_recv = [[ib_send_buffers[s][node_of(r)] for s in range(WORLD)]
37           for r in range(WORLD)]
38
39# ---- PHASE 2: NVLink fan-out to the exact owning GPU ----
40inbox = [[] for _ in range(WORLD)]
41for r in range(WORLD):
42    # Each rank now has bundles destined for *its node*. NVLink hop sorts
43    # them to the right GPU within the node.
44    for bundle in ib_recv[r]:
45        if bundle is None or len(bundle) == 0:
46            continue
47        # Re-route inside the node by recomputing topk on this small slice.
48        picks = route_topk(bundle, k)
49        for t in range(len(bundle)):
50            for e in picks[t]:
51                if node_of(owner(e)) == node_of(r):
52                    inbox[owner(e)].append((bundle[t], e))
53
54print("rank 0 inbox size:", len(inbox[0]))
55print("rank 6 inbox size:", len(inbox[6]))

Two observations. First, the IB phase does not need to know about the $G = 4$ GPUs per node — it only sees nodes. That collapse from $N \cdot G$ destinations to $N$ destinations is what lets NCCL pipeline IB more efficiently. Second, the NVLink phase is a small local all-to-all of bundles that are already on the right node. The node-limited router determines which bundles get sent, but the structure of the two-phase collective is independent of that.

PyTorch: Hierarchical all_to_all_single

In production this becomes two calls to torch.distributed.all_to_all_single, one on a cross-node process group and one on an intra-node group. The annotated code below is the skeleton; production implementations add capacity padding (so every collective transfers fixed-size buffers) and fuse the size-exchange into the dispatch itself.

1import torch
2import torch.distributed as dist
3
4def moe_dispatch_hierarchical(
5    x: torch.Tensor,          # (T, d) local tokens
6    expert_ids: torch.Tensor, # (T, k) chosen expert ids per token
7    *,
8    world_size: int,          # total ranks
9    gpus_per_node: int,
10    nodes_pg: dist.ProcessGroup,   # cross-node group: 1 rank per node-slot
11    intra_pg: dist.ProcessGroup,   # intra-node group: ranks within one node
12) -> tuple[torch.Tensor, torch.Tensor]:
13    """Two-phase MoE dispatch. Returns (recv_x, recv_expert_ids)."""
14
15    nodes = world_size // gpus_per_node
16    T, k = expert_ids.shape
17    d = x.shape[-1]
18    device = x.device
19    node_id = lambda eid: torch.div(eid, (expert_ids.numel() // nodes),
20                                    rounding_mode="floor")
21    target_nodes = node_id(expert_ids)            # (T, k)
22
23    # ---------- Phase 1: cross-node IB all-to-all ----------
24    # For each remote node, gather rows of x that need to visit it.
25    ib_send_chunks: list[torch.Tensor] = []
26    ib_send_sizes = torch.zeros(nodes, dtype=torch.long, device=device)
27    for n in range(nodes):
28        rows = (target_nodes == n).any(dim=1).nonzero(as_tuple=True)[0]
29        ib_send_chunks.append(x.index_select(0, rows))
30        ib_send_sizes[n] = rows.numel()
31
32    ib_send = torch.cat(ib_send_chunks, dim=0)    # (Σ rows, d)
33    ib_recv_sizes = torch.empty_like(ib_send_sizes)
34    dist.all_to_all_single(ib_recv_sizes, ib_send_sizes, group=nodes_pg)
35
36    ib_recv = torch.empty((int(ib_recv_sizes.sum().item()), d), device=device,
37                          dtype=x.dtype)
38    dist.all_to_all_single(
39        ib_recv, ib_send,
40        output_split_sizes=ib_recv_sizes.tolist(),
41        input_split_sizes=ib_send_sizes.tolist(),
42        group=nodes_pg,
43    )
44
45    # ---------- Phase 2: intra-node NVLink all-to-all ----------
46    # ib_recv now holds every token that needs an expert *on this node*.
47    # Bucket by exact owning GPU within the node.
48    local_target = (
49        node_id(expert_ids_for(ib_recv)) * 0 +    # placeholder; real code
50        gpu_within_node(ib_recv)                  # recomputes target ranks
51    )
52    # ... (length-collectives + dist.all_to_all_single on intra_pg) ...
53
54    return ib_recv, expert_ids  # simplified for clarity

NCCL note. In NCCL ≥ 2.18, all_to_all_single on a process group spanning a single node uses NVLink/NVSwitch directly. On a cross-node group it uses the IB NIC. By splitting the collective into two process groups, we let NCCL pick the right transport for each phase — and we expose the two phases to the overlap scheduler in DualPipe, which is the topic of the next section.

What Changes at 14.8T Tokens and 671B Parameters

Every quantity in the cost model scales with model and cluster size. At DeepSeek-V3 scale the relevant numbers are roughly:

Three things break or matter at this scale that did not in the 4-rank toy.

1. Capacity padding turns dynamic traffic into static

Real all_to_all_single calls are far cheaper when every rank knows the buffer sizes ahead of time — NCCL can skip the size-exchange and pre-allocate. Production EP rounds every per-expert token count up to a fixed capacity, drops the overflow, and pads the underflow with zeros. The system trades a small fraction of routed tokens for a deterministic, schedulable collective. The capacity factor (typically 1.25× the expected load) is one of the very few hyperparameters the operator has to tune after launch.

2. The number of expert-parallel ranks is bounded by the network

Above $N = 16$ nodes or so, even node-limited routing's constant $M$ hops begin to saturate IB on smaller clusters. DeepSeek-V3 keeps EP groups at one training sub-cluster (8 nodes = 64 GPUs), replicates that group via ZeRO data parallelism, and never grows EP horizontally. Scale comes from more DP replicas, not more EP nodes.

3. Failure modes are network-shaped

A flaky IB cable or a hot-spot port on a single node makes the EP all-to-all tail-latency-dominated. Because all ranks block on the same collective, one slow link freezes the whole step. Production clusters monitor per-link NIC counters and rebalance EP groups before the slow link becomes the global bottleneck.

Engineering Reality: Overlap, Topology, and Stragglers

Node-limited routing is necessary but not sufficient. Even with $M = 4$, IB time per MoE layer is still on the order of compute time per MoE layer. Doing nothing else means the cluster spends half its life on the network. The way DeepSeek-V3 closes the loop is by making the dispatch and combine overlap with the compute of another micro-batch.

• Compute-comm overlap. While micro-batch $m$ is running its MoE expert compute, the dispatch all-to-all for micro-batch $m + 1$ is already in flight on the NIC. The two queues run in parallel; the scheduler must be careful not to let IB drain before the next compute is ready.
• Topology-aware process groups. The cross-node group should map each node to one IB-attached rank that aggregates and forwards. Otherwise eight ranks per node fight for the same NIC and you lose the asymmetry advantage immediately.
• Straggler shielding. The slowest IB link in the cluster sets the wall time of the collective. Production clusters run weekly NIC linkspeed checks and quarantine nodes that drift.
• Dropless vs lossy capacity. "Dropless" EP, where overflow tokens are re-routed to spare experts, removes the quality penalty of capacity dropping but pushes more bytes on the wire. Pick lossy capacity in pre-training (sharper budgets, slight quality cost) and dropless in fine-tuning (where a single dropped token can pollute a small SFT dataset).