Implementing GRPO from Scratch

The Real Problem: PPO's Critic Is the Bottleneck

By the end of Section 15.2 we derived GRPO as PPO with the value head replaced by an in-group baseline. In Section 15.3 we pinned the hyperparameters DeepSeek used for R1. We now turn theory into code. The question is narrow and practical: given a rollout of G responses per prompt with a scalar reward each, what is the exact tensor program that updates the policy?

The four-model dance of PPO — policy, value head, reference, reward — is a memory monster at scale. At 70B parameters each model is roughly 140 GB in bf16. The value head alone, plus its gradients, plus its AdamW state, costs the same as a second policy replica. For DeepSeek V3 at 671B the value head was simply not affordable on the cluster they had. GRPO is the answer that fell out of that constraint, and its implementation is what makes the savings real.

Intuition: Compare Each Response to Its Siblings

Picture a class of four students answering the same hard math problem. Each turns in a different attempt. You grade them and obtain rewards $r_1, r_2, r_3, r_4$. How do you tell each student whether their answer was good?

You could grade against an absolute rubric — this is PPO with a learned value head, where the rubric is a neural net that predicts “what reward would I expect from a typical attempt at this problem?” The trouble is that the rubric itself has to be learned, and at 70B+ the rubric is as expensive as the student.

Or you can grade relative to the other students in the class — this is GRPO. The baseline for student $i$ is the average reward of the other three. If you outscored the class average you did well; if you fell below it you did poorly; if everyone scored the same, the class learned nothing from this problem and we move on.

The GRPO Objective in Equations

Given a prompt $q$ and a group of $G$ responses $\{o_1, \dots, o_G\}$ sampled from the old policy $\pi_{\theta_{\text{old}}}$, with scalar rewards $r_1, \dots, r_G$, the group-relative advantage of response $i$ is

$\quad A_i \;=\; \frac{r_i - \bar r}{\sigma_r}, \qquad \bar r = \frac{1}{G}\sum_{j=1}^G r_j, \qquad \sigma_r = \sqrt{\frac{1}{G}\sum_{j=1}^G (r_j - \bar r)^2}.$

Here $A_i$ is a SINGLE scalar broadcast to every token of $o_i$: for any token at position $t$ inside response $o_i$, the advantage used in the gradient is $A_{i,t} = A_i$. There is no per-token credit assignment beyond what the PPO clip already does.

The per-token probability ratio is the standard off-policy correction:

$\quad \rho_{i,t}(\theta) \;=\; \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})}.$

The clipped surrogate is PPO's, applied per token:

$\quad L_i^{\text{surr}}(\theta) \;=\; \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min\!\big( \rho_{i,t}\,A_i, \; \text{clip}(\rho_{i,t},\,1-\varepsilon,\,1+\varepsilon)\,A_i \big).$

The per-token KL penalty against the frozen reference policy uses Schulman's $k_3$ estimator, which is unbiased and sample-wise non-negative:

$\quad \widehat{\mathrm{KL}}_{i,t} \;=\; \exp\!\big( \log \pi_{\text{ref}} - \log \pi_\theta \big) \;-\; \big( \log \pi_{\text{ref}} - \log \pi_\theta \big) \;-\; 1.$

Putting it together, vanilla GRPO maximises (equivalently, minimises the negative of)

$\quad \mathcal{J}_{\text{GRPO}}(\theta) \;=\; \frac{1}{G} \sum_{i=1}^{G} \Big[ L_i^{\text{surr}}(\theta) \;-\; \beta \cdot \tfrac{1}{|o_i|} \sum_{t=1}^{|o_i|} \widehat{\mathrm{KL}}_{i,t} \Big].$

Symbols, in order: $q$ the prompt; $G$ the group size; $o_i$ the $i$-th sampled response; $r_i$ its scalar reward; $A_i$ its group-relative advantage; $\rho_{i,t}$ the per-token importance ratio of the live vs. old policy; $\varepsilon$ the PPO clip range (0.2 in DeepSeek R1); $\pi_{\text{ref}}$ the frozen reference (SFT) policy; $\beta$ the per-token KL coefficient (0.04 in DeepSeek R1).

Manual Numerical Walkthrough: One GRPO Update

We trace one GRPO step on a single prompt with $G = 4$ short responses. Every intermediate number is shown; the toy is set up so that no clipping happens for most tokens and exactly one clip event happens for response 3.

Step 1 — rewards and group statistics

Rewards from the reward function: $r = (0.9,\, 0.3,\, -0.1,\, 0.7)$. Group mean and group standard deviation:

$\bar r = (0.9 + 0.3 - 0.1 + 0.7)/4 = 0.45,$ $\sigma_r = \sqrt{(0.45^2 + 0.15^2 + 0.55^2 + 0.25^2)/4} = \sqrt{0.1475} \approx 0.384.$

Step 2 — per-response advantages

Notice the advantages sum to zero by construction. The most-rewarded response ($o_1$) gets the largest positive advantage; the least-rewarded ($o_3$) gets the largest negative advantage.

Step 3 — per-token ratios for one response

Take response 3, which has 2 tokens. Suppose the log-ratios (live vs old) are $\log\rho_{3,1} = +0.25,\;\log\rho_{3,2} = -0.30$. Exponentiating: $\rho_{3,1} \approx 1.284,\;\rho_{3,2} \approx 0.741$. Both lie OUTSIDE the trust region $[1-\varepsilon,\,1+\varepsilon] = [0.8,\,1.2]$.

Step 4 — the asymmetric clip in action

With $A_3 = -1.432$:

The mean objective for response 3 is $(-1.838 - 1.146)/2 = -1.492$. The clip did exactly what it should: it allowed the full penalty when the live policy over-assigned probability to a bad token ($t=1$), but prevented an outsized penalty when the live policy was already moving away from the bad token ($t=2$).

Step 5 — per-token KL (Schulman k3)

For response 1, token 1, suppose $\log\pi_\theta = -0.476,\;\log\pi_{\text{ref}} = -0.600$. Then $\log(\pi_{\text{ref}}/\pi_\theta) = -0.124$ and

$\widehat{\mathrm{KL}} = e^{-0.124} - (-0.124) - 1 = 0.8834 + 0.124 - 1 = 0.0074.$

Small, positive — the policy has drifted only slightly from the reference. Multiplied by $\beta = 0.04$ the KL penalty per token is $\approx 0.0003$ — a gentle leash, not a tight one.

Step 6 — aggregate into one scalar loss

Compute the per-response mean objective for each of the four responses (with ratios mostly near 1), negate, add $\beta \cdot \overline{\mathrm{KL}}$, and average over the group. With ratios this close to 1 the loss is small — the policy will only start moving once enough gradient steps have shifted the per-token log-probs meaningfully.

Sanity reading of the numbers: on a fresh rollout every ratio is very close to 1, every KL term is close to 0, and the loss is small. That is healthy. A first-step loss with $|\rho - 1| > 0.1$ typically means the rollout and the trainer are computing log-probs differently — check sampling temperature, dtype, and the prompt/response mask alignment.

Interactive: The Group-Relative Advantage

Drag the reward sliders below to see how a group of four responses gets turned into four advantages. Notice three things in particular:

1. If you align all four rewards to the same value every advantage collapses to zero. This group contributes nothing to the gradient — GRPO learns from disagreement inside a group, not from absolute reward.
2. Shifting all four rewards up or down by the same amount leaves the advantages unchanged. Only the spread inside the group matters.
3. Toggle off “divide by std” to see the Dr.GRPO variant (Section 15.5): the advantages now scale with the spread, which avoids vanilla GRPO's subtle bias toward low-variance groups.

Plain Python: GRPO Loss from Scratch

We re-implement the GRPO objective in pure NumPy on the toy rollout from the walkthrough above. No autograd, no transformer — just the loss arithmetic. Swap the hard-coded logits for a real transformer's output and the code is unchanged.

G = 4, response lengths = [3, 2, 2, 2], total response tokens = 9

ref_logp[0] = [-0.60, -0.71, -0.82] (one entry per response-1 token)

rewards = [0.9, 0.3, -0.1, 0.7] — one scalar per response, not per token

mean_r = (0.9 + 0.3 - 0.1 + 0.7) / 4 = 0.45; std_r = 0.384

A_per_response = [(0.9-0.45)/0.384, (0.3-0.45)/0.384, (-0.1-0.45)/0.384, (0.7-0.45)/0.384] = [+1.172, -0.391, -1.432, +0.651]

new_logits[0].shape = (3, 3) — three tokens, three-action vocab

logits row [1.10, -0.20, 0.30] → log-probs row [-0.476, -1.776, -1.276] (sum-exp = 1 by construction)

actions[0] = [0, 2, 1]  →  new_logp_i = [log_pall[0,0], log_pall[1,2], log_pall[2,1]] = [-0.476, -0.176, -0.376]

old_logp[0][0] = -0.50, new_logp_i[0] = -0.476  →  log_ratio[0] = 0.024  →  ratio[0] = exp(0.024) = 1.024

For o_1 (A=+1.172): ratio=[1.024, 1.105, 1.052]  →  surr1=[1.200, 1.295, 1.232]; all ratios inside [0.8, 1.2] so surr2 = surr1; obj = surr1.

For o_3 (A=-1.432) with ratio=[1.284, 0.741]: surr1=[-1.838, -1.061]; surr2=[1.2·-1.432, 0.8·-1.432]=[-1.718, -1.146]; min = [-1.838, -1.146]. Token 0 is unclipped (A<0, r>1+ε → full penalty); token 1 is clipped (A<0, r<1-ε → bounded penalty).

log_new_i[0] = -0.476, log_ref_i[0] = -0.60  →  log_r_ref = -0.124  →  k3 = exp(-0.124) - (-0.124) - 1 = 0.0078

For o_1: obj.mean() = (1.200+1.295+1.232)/3 = 1.242; kl_tok.mean() ≈ 0.0035  →  per_response_loss[0] = -1.242 + 0.04 * 0.0035 = -1.242

per_response_loss = [-1.242, +0.402, +1.492, -0.651]  →  loss = mean ≈ 0.000 (advantages cancel by construction, ratios are ~1 → tiny first-step loss is expected)

1"""
2GRPO objective — pure NumPy, no autograd, no value head.
3
4For one PROMPT we collected a GROUP of G responses with the OLD policy.
5Each response has been scored by a REWARD function (rule-based, judge,
6or learned RM) into a single scalar. That scalar advantage will be
7broadcast over every token of that response.
8
9The 'policy' here is a single-token categorical over a 3-action vocab
10so every number is readable. Swap in a transformer logits head and
11the loss code is byte-for-byte what GRPO trainers use in production.
12"""
13
14import numpy as np
15
16# ---------------------------------------------------------------------------
17# 1. One prompt, G = 4 responses sampled from the OLD policy.
18#    Each response is a token sequence; rewards is one scalar per response.
19# ---------------------------------------------------------------------------
20
21#               o_1            o_2          o_3       o_4
22actions = [[0, 2, 1],       [1, 0],      [2, 2],   [0, 1]]
23old_logp = [
24    [-0.50, -0.61, -0.92],  [-1.20, -0.41],  [-0.81, -0.69], [-0.51, -1.10],
25]
26ref_logp = [
27    [-0.60, -0.71, -0.82],  [-1.10, -0.51],  [-0.91, -0.79], [-0.41, -1.00],
28]
29rewards = np.array([0.9, 0.3, -0.1, 0.7])   # one scalar per response
30G = len(rewards)
31
32# ---------------------------------------------------------------------------
33# 2. Group-relative advantage — GRPO's signature move.
34#    Replace V(s) with the in-group mean and std of rewards.
35#    A_i is the SAME number for every token of response i.
36# ---------------------------------------------------------------------------
37
38mean_r = rewards.mean()
39std_r  = rewards.std() + 1e-8
40A_per_response = (rewards - mean_r) / std_r          # shape (G,)
41
42# ---------------------------------------------------------------------------
43# 3. The NEW (live) policy's logits at the SAME states. In a real run we
44#    re-forward each response's tokens through the live model after every
45#    gradient step. Here we hard-code plausible "slightly drifted" logits.
46# ---------------------------------------------------------------------------
47
48new_logits = [
49    np.array([[ 1.10, -0.20,  0.30],     # o_1, token 0
50              [-0.40,  0.80,  1.60],     # o_1, token 1
51              [ 0.10,  1.40,  0.20]]),   # o_1, token 2
52    np.array([[-0.20,  1.10,  0.40],     # o_2, token 0
53              [ 0.90, -0.30,  0.10]]),   # o_2, token 1
54    np.array([[ 0.10,  0.20,  1.30],     # o_3, token 0
55              [-0.10,  0.40,  1.50]]),   # o_3, token 1
56    np.array([[ 1.20, -0.40,  0.10],     # o_4, token 0
57              [-0.30,  1.50,  0.20]]),   # o_4, token 1
58]
59
60# ---------------------------------------------------------------------------
61# 4. Per-token GRPO terms for each response.
62# ---------------------------------------------------------------------------
63
64EPSILON   = 0.2          # PPO-style clip range
65BETA_KL   = 0.04         # per-token KL coefficient (DeepSeek R1 default)
66
67per_response_loss = []
68per_response_kl   = []
69
70for i in range(G):
71    # log pi_new at each token via stable log-softmax
72    z         = new_logits[i] - new_logits[i].max(axis=-1, keepdims=True)
73    log_pall  = z - np.log(np.exp(z).sum(axis=-1, keepdims=True))
74    new_logp_i = log_pall[np.arange(len(actions[i])), actions[i]]   # (T_i,)
75
76    old_logp_i = np.array(old_logp[i])
77    ref_logp_i = np.array(ref_logp[i])
78
79    # Probability ratio in log space, then exponentiate.
80    log_ratio = new_logp_i - old_logp_i
81    ratio     = np.exp(log_ratio)
82
83    # PPO clipped surrogate — A is the SAME scalar across all tokens of o_i.
84    A         = A_per_response[i]
85    surr1     = ratio * A
86    surr2     = np.clip(ratio, 1.0 - EPSILON, 1.0 + EPSILON) * A
87    obj       = np.minimum(surr1, surr2)
88
89    # Schulman k3 unbiased KL estimator vs the REFERENCE policy.
90    # k3 = exp(log_ref - log_new) - (log_ref - log_new) - 1   >= 0
91    log_r_ref = ref_logp_i - new_logp_i
92    kl_tok    = np.exp(log_r_ref) - log_r_ref - 1.0
93
94    # Per-RESPONSE mean: equal weight to every token of o_i.
95    per_response_loss.append(-obj.mean() + BETA_KL * kl_tok.mean())
96    per_response_kl.append(kl_tok.mean())
97
98# ---------------------------------------------------------------------------
99# 5. Final scalar loss = uniform mean over the GROUP of responses.
100#    (Vanilla GRPO — every response gets the same weight regardless of
101#    length. Dr.GRPO removes the per-response length normalisation; we
102#    show that variant in the engineering-reality section below.)
103# ---------------------------------------------------------------------------
104
105loss = np.mean(per_response_loss)
106
107print(f"advantages   = {A_per_response.round(3)}")
108print(f"per-resp KL  = {[round(k, 4) for k in per_response_kl]}")
109print(f"loss         = {loss:.4f}")

PyTorch: A Production-Shaped GRPO Step

Now the full thing: tensorised across the (B, G, S) layout that every production GRPO trainer uses, with the group-relative normalisation as a single mean/std along the G axis, gradient clipping, and the diagnostics that go straight onto a training dashboard. Drop in any HuggingFace causal LM for policy and a frozen copy of the SFT checkpoint for ref_policy and this trains a real reasoning model.

B = 8 prompts, G = 16 responses, S = 1024 tokens  →  input_ids.shape = (8, 16, 1024); rewards.shape = (8, 16)

rewards.shape = (8, 16); grp_mean.shape = (8, 1)  →  prompt 0's baseline is the mean of its own 16 responses

For a math prompt where 16 of 16 responses are wrong (reward=0): mean=0, std≤1e-8, every advantage ≈ 0, the group contributes no gradient (as it should — there is nothing to learn from)

adv before: shape (8, 16); after unsqueeze + expand: shape (8, 16, 1024); adv[0, 3, :] is one repeated scalar = (rewards[0,3] - grp_mean[0]) / grp_std[0]

input_ids.view(8*16, 1024) = (128, 1024)  →  out.logits = (128, 1024, 128256)  →  reshape to (8, 16, 1023, 128256)

Typical first-step values: log_ratio in [-0.05, +0.05]  →  ratio in [0.951, 1.051]

Worked example: A = +1.172, ratio = 1.105 (in range)  →  surr1 = 1.295, surr2 = 1.295, obj = 1.295.  A = -1.432, ratio = 1.284 (above +ε)  →  surr1 = -1.838, surr2 = -1.718, obj = min = -1.838 (full penalty, unclipped)

If log_ref = -0.60, log_new = -0.476 (model slightly more confident than reference): log_r_ref = -0.124  →  k3 = exp(-0.124) - (-0.124) - 1 = 0.883 + 0.124 - 1 = 0.0073

For a response of true length 87 inside an S=1024 padded slot: tok_count = 87 (not 1024); obj_per_resp = (obj * am).sum() / 87

Typical first-step values: policy_loss ≈ -0.05, kl_loss ≈ 0.001  →  loss ≈ -0.05 + 0.04*0.001 = -0.04996

1"""
2PyTorch GRPO step — the inner loop of every modern verifier-based RL
3trainer (DeepSeek R1, Olmo 3, Qwen-Math-RL, AceMath-RL).
4
5A rollout is a list of PROMPTS, each rolled out to G responses with
6the OLD policy (frozen snapshot). For each token of each response we have:
7   - old_log_prob   : log pi_old(a_t | s_t)            (no gradient)
8   - ref_log_prob   : log pi_ref(a_t | s_t)            (no gradient)
9For each response we have:
10   - reward         : scalar from the reward function
11GRPO turns rewards into advantages by group-mean / group-std within each
12prompt's group, then runs PPO's clipped surrogate token-wise.
13"""
14
15import torch
16import torch.nn.functional as F
17
18EPSILON   = 0.2          # PPO clip range  (DeepSeek R1: 0.2 lo, 0.28 hi)
19BETA_KL   = 0.04         # per-token KL to reference policy
20MAX_GRAD  = 1.0          # global grad-norm clip
21
22def grpo_step(batch, policy, ref_policy, optimizer):
23    """
24    batch contains tensors laid out as (B, G, S) where:
25        B = number of prompts in the minibatch
26        G = number of sampled responses per prompt
27        S = max response length (right-padded with zeros)
28
29    Fields:
30        input_ids       : (B, G, S)   prompt + response tokens
31        attn_mask       : (B, G, S)
32        action_mask     : (B, G, S)   1 on RESPONSE tokens, 0 on prompt/pad
33        old_logp        : (B, G, S)   log pi_old at each token
34        ref_logp        : (B, G, S)   log pi_ref at each token
35        rewards         : (B, G)      one scalar per response
36    """
37    input_ids   = batch["input_ids"]
38    attn_mask   = batch["attn_mask"]
39    action_mask = batch["action_mask"]
40    old_logp    = batch["old_logp"]
41    ref_logp    = batch["ref_logp"]
42    rewards     = batch["rewards"]
43
44    B, G, S = input_ids.shape
45
46    # ---- 1. Group-relative advantage ------------------------------------
47    # Mean and std are computed PER PROMPT, across the G responses.
48    grp_mean = rewards.mean(dim=1, keepdim=True)                # (B, 1)
49    grp_std  = rewards.std (dim=1, keepdim=True).clamp(min=1e-8)
50    adv      = (rewards - grp_mean) / grp_std                   # (B, G)
51    adv      = adv.unsqueeze(-1).expand(B, G, S)                # broadcast
52
53    # ---- 2. Forward the LIVE policy on the response tokens --------------
54    # Flatten the (B, G) groups into a single batch for the forward pass.
55    ids_flat = input_ids.view(B * G, S)
56    am_flat  = attn_mask.view(B * G, S)
57    out      = policy(input_ids=ids_flat, attention_mask=am_flat,
58                      use_cache=False)
59    logits   = out.logits[:, :-1, :].view(B, G, S - 1, -1)      # predict t+1
60    targets  = input_ids[:, :, 1:]                              # (B, G, S-1)
61    am       = action_mask[:, :, 1:].float()
62    adv      = adv[:, :, 1:]
63
64    log_probs_all = F.log_softmax(logits, dim=-1)
65    new_logp = log_probs_all.gather(-1, targets.unsqueeze(-1)).squeeze(-1)
66
67    # ---- 3. Clipped surrogate -------------------------------------------
68    log_ratio = new_logp - old_logp[:, :, 1:]
69    ratio     = log_ratio.exp()
70
71    surr1 = ratio * adv
72    surr2 = torch.clamp(ratio, 1 - EPSILON, 1 + EPSILON) * adv
73    obj   = torch.min(surr1, surr2)
74
75    # ---- 4. Per-token KL to the reference policy (k3) -------------------
76    log_r_ref = ref_logp[:, :, 1:] - new_logp                   # log(ref/new)
77    kl_tok    = log_r_ref.exp() - log_r_ref - 1.0               # >= 0
78
79    # ---- 5. Per-response means, then GROUP mean -------------------------
80    # Sum within each response, divide by the response's token count.
81    tok_count = am.sum(dim=-1).clamp(min=1.0)                   # (B, G)
82    obj_per_resp = (obj * am).sum(dim=-1) / tok_count           # (B, G)
83    kl_per_resp  = (kl_tok * am).sum(dim=-1) / tok_count        # (B, G)
84
85    # Vanilla GRPO: uniform mean across the G responses, then across B.
86    policy_loss = -(obj_per_resp).mean()
87    kl_loss     =  (kl_per_resp).mean()
88
89    loss = policy_loss + BETA_KL * kl_loss
90
91    # ---- 6. One gradient step -------------------------------------------
92    loss.backward()
93    gn = torch.nn.utils.clip_grad_norm_(policy.parameters(), MAX_GRAD)
94    optimizer.step()
95    optimizer.zero_grad(set_to_none=True)
96
97    # ---- 7. Diagnostics --------------------------------------------------
98    with torch.no_grad():
99        approx_kl = ((old_logp[:, :, 1:] - new_logp) * am).sum() / am.sum()
100        clipfrac  = (((ratio - 1.0).abs() > EPSILON).float() * am).sum() / am.sum()
101        adv_std   = adv[am.bool()].std()
102
103    return {
104        "loss":       loss.detach(),
105        "pg_loss":    policy_loss.detach(),
106        "kl_ref":     kl_loss.detach(),
107        "approx_kl":  approx_kl.detach(),
108        "clipfrac":   clipfrac.detach(),
109        "adv_std":    adv_std.detach(),
110        "grad_norm":  gn.detach(),
111    }

At Massive Scale: What Changes for a 70B+ Run

Every line of code above scales to a 70B-parameter GRPO run, but three things change in spirit.

Memory: GRPO's biggest gift to production training

At 70B parameters in bf16, the policy and the reference together are ~280 GB of weights. Adding a value head (PPO) doubles weights to ~420 GB, plus another ~140 GB of value-head gradients, plus ~280 GB of AdamW optimiser state for the value head — an extra ~560 GB relative to GRPO. On an 8×H100 node with 80 GB per GPU, that extra 560 GB is the difference between a run that fits with FSDP sharding and a run that requires a second node just for the value head. This is why DeepSeek picked GRPO for R1 and why every post-R1 reasoning recipe (Qwen-Math-RL, AceMath-RL, Olmo 3) followed suit.

The (B, G) axis lives across ranks

At scale, the natural FSDP sharding is to split $B \cdot G$ across data-parallel ranks (every rank holds the same parameters and a different slice of prompts) and shard the parameters across model-parallel ranks. The group-relative mean/std becomes a careful primitive: each rank computes partial sum and partial sum-of-squares along the G axis for its own prompts; an all-reduce is NOT needed for the mean and std as long as every prompt's entire group is on the same rank. The most common GRPO bug at scale is splitting a single prompt's group across ranks — each rank then sees a different baseline, the effective advantage scale varies across ranks, and the run becomes noisy in a way that takes a week to diagnose. Every production GRPO library has unit tests that assert all G responses for a prompt live on the same rank.

Sampling-vs-training imbalance is even worse than PPO

GRPO requires $G$× more rollout tokens per prompt than PPO did. With $G = 16$ and responses of length 1024 tokens that is 16 384 tokens per prompt to sample before a single gradient step can happen. At 500 tok/s/GPU with vLLM, even a 1024-prompt minibatch takes minutes to roll out. Production GRPO infrastructure runs the rollout cluster (many GPUs, prioritised for inference throughput with vLLM and continuous batching) entirely separately from the training cluster (FSDP, fewer GPUs, prioritised for memory bandwidth) and ships generations over the network. Weights sync after each optimisation phase.

Reward model bottleneck disappears for verifier rewards

DeepSeek R1 uses RULE-BASED rewards for math and code (the verifier is the unit test or the symbolic check), which is essentially free to compute. For tasks that still need a learned RM, the same rollout/score/train split as PPO applies, and the RM becomes the new cost bottleneck.

Engineering Reality: The Knobs That Break Runs

After enough GRPO runs every team converges on the same set of failure modes and the same set of high-leverage fixes.

• All-equal-reward groups dominate the batch. Symptom: loss is flat, gradient norm collapses, advantage std drops near zero. Cause: for hard math/code prompts every response in a group is wrong (reward = 0) so every advantage is zero. The group contributes nothing. Fix: oversample these prompts with higher temperature, or skip them entirely from the gradient and log them separately for later curriculum work. DeepSeek R1 explicitly filters such groups from the loss.
• Group split across ranks. Symptom: loss noise is much higher than a single-GPU debug run. Cause: a careless DataLoader put 8 of a prompt's 16 responses on rank 0 and 8 on rank 1. The baseline computed on rank 0 differs from rank 1 and the advantages are inconsistent. Fix: a custom sampler that guarantees prompt_id % num_ranks == rank_id for the entire group. This is a one-time setup that prevents weeks of pain.
• Length bias toward longer responses. Symptom: mean response length climbs steadily over training even though rewards do not require length. Cause: vanilla GRPO's per-response $1/|o_i|$ normalisation gives long responses a smaller per-token loss; the policy can game this by lengthening responses to dilute the gradient. Fix: switch to Dr.GRPO's $1/S_{\max}$ normalisation (Section 15.5), or apply an explicit length penalty inside the reward function.
• KL coefficient too small. Symptom: reward climbs for 200 steps then the policy degenerates (repeats a single phrase, drops English, emits gibberish that scores high on a mis-calibrated RM). Cause: insufficient KL pressure against the reference SFT model. Fix: raise $\beta$ by 2× and restart from a pre-collapse checkpoint. Many teams use adaptive KL (raise $\beta$ when realised KL exceeds a target, lower it when below).
• Rollout temperature mismatch. Symptom: ratio distribution is centred far from 1.0 on step 0 even with the freshly-cloned policy. Cause: the rollout sampled with temperature=1.0 but the trainer computes $\log\pi_{\text{new}}$ from $\text{softmax}(\text{logits})$. The two distributions are different and the ratio is meaningless. Fix: apply the SAME temperature inside the trainer when computing per-token log-probs, or sample with temperature=1.0 throughout.
• Action mask off by one. Symptom: training looks fine, reward inches up, but eval shows no change from the SFT model. Cause: the mask includes prompt tokens (which carry no gradient by construction since their log-probs are not part of the decision), or excludes the first response token. Fix: print one example's mask alongside its decoded tokens before launch and eyeball that mask=1 lines up with the response tokens only. The off-by-one between input_ids[:, :, :-1] and input_ids[:, :, 1:] is the most common variant.
• Old log-probs not refreshed between PPO epochs. Symptom: ratio drifts far from 1 after the second optimisation epoch on the same rollout, clipfrac spikes. Cause: GRPO typically uses 1 epoch per rollout for exactly this reason, but if you set PPO-style 4 epochs you must NOT re-use the old_logp from epoch 1 on epoch 4 — the live policy has moved. Fix: either stick with 1 epoch (the DeepSeek R1 choice) or refresh old_logp at the start of every epoch.
• Gradient norm explodes after a few hundred steps. Symptom: $|g|$ climbs from 0.5 to 50 in 100 steps. Cause: either KL coefficient too small (policy escaping to extreme outputs) or a reward model bug that gives unboundedly large rewards on some inputs. Fix: raise $\beta$ or clip the reward distribution at the source.
• Reference accidentally on the same fork as policy. Symptom: KL goes to zero, policy never moves. Cause: the “reference” was initialised as a Python reference to the same tensors as the live policy. Fix: assert id(ref_policy) != id(policy) at startup; load the reference from a separate checkpoint.

The Mental Model That Unifies This Section

GRPO is PPO with one substitution and two deletions. Substitution: replace $A_t = R_t - V(s_t)$ with the group-normalised reward $A_i = (r_i - \bar r)/\sigma_r$, broadcast over every token of response $i$. Deletion 1: remove the value head, its gradient, and its optimiser state (saving ~3 model-replicas worth of memory at scale). Deletion 2: remove the entropy bonus (the per-group reward diversity replaces it as an exploration signal). Everything else — the clipped surrogate, the per-token KL to the reference, the gradient clip, the diagnostics — is the PPO code unchanged. Get the (B, G, S) shape right, keep every group on one rank, and a 70B reasoning model can be RL-trained for a thousand steps on a single 8×H100 node.

With this section closed, Chapter 15 is complete. The next chapter opens DeepSeek R1-Zero — the pure-RL experiment that ran this exact GRPO loop with rule-based math rewards and discovered emergent chain-of-thought reasoning along the way.