GRPO Hyperparameters from DeepSeek R1

Section 15.2 derived GRPO from PPO by eliminating the critic and replacing the value-baseline with a per-prompt z-score over a group of sampled completions. That derivation gives you a loss function with four exposed knobs: the group size $G$, the PPO clip range $\epsilon$, the KL coefficient $\beta$, and the learning rate. The algorithm is now exactly as good as the numbers you put into those knobs. DeepSeek R1 picked a specific combination — most of which is unusual compared to classic PPO defaults — and that combination is much of what made the run successful. This section is about why those numbers.

The thesis of this section. GRPO is not a single algorithm, it is a family of algorithms parameterised by $(G, \epsilon, \beta, \eta)$. DeepSeek R1 sits at $G = 16$, $\epsilon = 0.2$, $\beta = 10^{-3}$, $\eta = 10^{-6}$. Each of those values trades a specific failure mode against a specific cost, and the four together define a stable corridor in which long-chain reasoning can emerge from a rule-based reward.

The Real Problem: Hyperparameters Are the Whole Algorithm

Most papers describe an algorithm in equations and bury the hyperparameters in an appendix. For RL fine-tuning of large language models, that ordering is wrong. The equations are common to PPO, GRPO, and their cousins; the numbers determine whether the run converges to a frontier reasoning model or collapses to repetitive nonsense within a few hundred steps. The R1 release made this explicit: the supplementary material is dominated not by new math but by the exact numerical settings of an otherwise standard GRPO loop. Reproducing R1 means matching the numbers, not just the symbols.

Three categories of catastrophe lurk at the boundaries:

Every one of these failure modes is visible in the loss curve within a few hundred steps if you know which curve to watch. The R1 hyperparameter choices were calibrated to keep the run inside a narrow band where none of them fires. Understanding those choices is the difference between a paper you can read and a paper you can reproduce.

Intuition: Four Knobs That Move Together

Think of GRPO as a sailing problem. You are steering a boat (the policy) across a noisy sea (sampled rewards) toward an island (the reasoning behaviour you want to reinforce). The four hyperparameters are four physical levers on the boat, and they have to be balanced:

1. $G$ is your sample size against the waves. Each prompt generates $G$ trajectories. The mean and std of their rewards is the signal you steer by. Bigger $G$ means less noise but more forward-pass cost. $G = 16$ is the sweet spot where the std estimate stabilises at order-of-magnitude accuracy without burning four times the compute that $G = 64$ would.
2. $\epsilon$ is how hard you are allowed to push the rudder in one step. The PPO clip forces each per-token policy ratio to stay inside $(1 - \epsilon, 1 + \epsilon)$. Bigger $\epsilon$ lets the policy move further per update but risks instability; smaller $\epsilon$ is safe but slow. $\epsilon = 0.2$ is the literal copy-paste from PPO — it works because GRPO uses only one optimisation epoch per rollout, so the ratio rarely walks far enough to need a tighter clip.
3. $\beta$ is the rope tethering the boat to its starting point. The KL penalty pulls the policy toward the reference (post-SFT) model. Heavy $\beta$ keeps you safe but prevents you from discovering anything the reference cannot already do. R1 uses a deliberately tiny $\beta = 10^{-3}$ — almost no tether — so the policy is free to invent the long-chain reasoning behaviours that the SFT base would never have produced.
4. $\eta$ is the throttle. Learning rate $10^{-6}$ is two orders of magnitude smaller than the SFT learning rate, because every RL gradient is amplified by a Monte-Carlo estimator with variance $\sigma^2 / G$. A bigger throttle on a noisier engine is how you tear the gearbox apart.

The Mathematical Form of the GRPO Update

Recall the GRPO objective derived in Section 15.2. For a prompt $x$ and a group of $G$ sampled completions $\{y_i\}_{i=1}^{G}$ with rewards $r_i$, the per-prompt objective is:

$J_{\text{GRPO}}(\theta) = \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \left[ \min(\rho_{i,t} A_i, \; \text{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon) A_i) - \beta \, \widehat{\text{KL}}_{i,t} \right]$

where each symbol carries one of the hyperparameters:

The advantage with no critic

The single line that defines GRPO is $A_i = (r_i - \mu_g) / \sigma_g$ where $\mu_g, \sigma_g$ are computed over the $G$ completions of one prompt. There is no value head, no GAE, no Monte-Carlo return — just the empirical mean and standard deviation of the group's rewards. The variance of this estimator is $\text{Var}(\mu_g) \approx \sigma_r^2 / G$, which is exactly why $G$ sits at the centre of every hyperparameter discussion.

The clip with one optimisation epoch

Classic PPO uses 4–10 epochs over each rollout, which lets the ratio $\rho$ drift several times and makes a tight clip essential. GRPO as run in R1 uses one epoch — one optimiser step per rollout — so the ratio rarely wanders. That choice changes the role of $\epsilon$: it is no longer the binding constraint, it is a safety rail. R1's $\epsilon = 0.2$ is therefore the inherited PPO default; DAPO, which uses multiple epochs and asymmetric clipping, has to set $\epsilon_{\text{high}} = 0.28$ and $\epsilon_{\text{low}} = 0.2$ for the same reason classic PPO needs a tight clip.

The KL estimator and why beta is so small

DeepSeek's unbiased estimator $\widehat{\text{KL}} = e^{r} - r - 1$ with $r = \log \pi_{\text{ref}} - \log \pi_{\theta}$ is non-negative by construction (it is a Bregman divergence) and zero at agreement. With $\beta = 10^{-3}$, even a large per-token KL of $0.05$ contributes only $5 \cdot 10^{-5}$ to the objective — three orders of magnitude smaller than a typical surrogate term of $\sim 0.05$. The KL is therefore decorative early on. It bites only after thousands of steps when the policy has wandered far enough that per-token KL sits in the $0.5 - 2.0$ range; at that point the integrated penalty becomes the dominant force preventing collapse.

The DeepSeek R1 Recipe

The published numbers, in one table:

Three of these are unusual; the rest are PPO defaults. The unusual ones are $G = 16$ (PPO classically uses $G = 4$), $\beta = 10^{-3}$ (PPO classically uses $0.04$), and one PPO epoch (PPO classically uses 4–10). Together these three changes encode the entire GRPO philosophy: trade off-policy correction for sample-mean variance reduction, and trade KL safety for exploration freedom.

Manual Numerical Walkthrough

One prompt, four completions, every number computed by hand.

Visualizing the Hyperparameter Regimes

The widget below runs the same GRPO update for one prompt under four preset hyperparameter regimes — DeepSeek R1, DAPO, classic-PPO defaults, and an over-regularised collapse — plus the ability to drag $G$, $\epsilon$, and $\beta$ continuously and watch the per-completion bars rearrange.

Four movements worth performing in the widget. First, drag $G$ from 16 down to 2 — the std of the group rewards becomes unstable and the advantages start swinging wildly between resamples (use the seed slider). Second, drag $\epsilon$ from 0.2 down to 0.05 — the "Fraction clipped" card climbs and the mean surrogate flattens, because the policy is forbidden from moving. Third, drag $\beta$ from 0.001 up to 0.1 — the sky-blue KL overlay starts eating the green surrogate bars from the right; the policy is being pulled back to the reference faster than the reward signal can push it forward. Fourth, click the "Classic PPO defaults" preset and observe that both small G and large beta are applied at once — that is the regime in which a vanilla PPO loop would refuse to discover R1-style reasoning, and it is precisely what DeepSeek had to walk away from.

Plain Python: One GRPO Update Step

The plain-Python version below performs one full GRPO update step on a single prompt with $G = 16$ completions, using only the standard library. No PyTorch, no batching, no GPU — just the arithmetic that the R1 training loop executes for every prompt in every batch.

1import math
2import random
3
4# DeepSeek R1 GRPO hyperparameters (publicly reported)
5G        = 16          # group size: completions per prompt
6EPSILON  = 0.2         # PPO clip range
7BETA     = 0.001       # KL coefficient (very small on purpose)
8LR       = 1e-6        # AdamW learning rate
9TEMP     = 1.0         # sampling temperature during rollout
10MAX_LEN  = 32768       # max generation length in tokens
11
12# One math question; assume we already sampled G completions and graded them
13# with a rule-based reward in {0, 1} (1 = correct, 0 = wrong).
14random.seed(0)
15rewards = [random.random() < 0.55 for _ in range(G)]
16rewards = [1.0 if r else 0.0 for r in rewards]
17
18# Step 1: group-normalised advantages. No critic; just a z-score over the group.
19mu   = sum(rewards) / G
20var  = sum((r - mu) ** 2 for r in rewards) / G
21sd   = math.sqrt(var) + 1e-8
22A    = [(r - mu) / sd for r in rewards]
23
24# Step 2: per-token log-prob ratios from the old and new policies.
25# In a real run these come from running both models on the rollout. Here we
26# fake a tiny shift so the clip mechanism is visible.
27log_ratios = [random.gauss(0.0, 0.1) for _ in range(G)]
28rho        = [math.exp(lr) for lr in log_ratios]
29
30# Step 3: PPO-style clipped surrogate, per completion.
31def surrogate(rho_i, A_i):
32    unclipped = rho_i * A_i
33    clipped   = max(min(rho_i, 1 + EPSILON), 1 - EPSILON) * A_i
34    return min(unclipped, clipped)
35
36surr = [surrogate(rho[i], A[i]) for i in range(G)]
37
38# Step 4: unbiased KL estimator from the GRPO paper, KL >= 0, zero at rho=1.
39kl   = [r - math.log(r) - 1 for r in rho]
40
41# Step 5: GRPO objective per completion = surrogate - beta * KL. Mean over group.
42J    = sum(surr[i] - BETA * kl[i] for i in range(G)) / G
43
44clipped_frac = sum(1 for i in range(G)
45                   if rho[i] != max(min(rho[i], 1 + EPSILON), 1 - EPSILON)
46                   and A[i] != 0) / G
47
48print(f"Group reward mean / std   : {mu:.3f} / {sd:.3f}")
49print(f"Mean surrogate            : {sum(surr)/G:+.4f}")
50print(f"Mean KL drag (beta * KL)  : {BETA * sum(kl)/G:.6f}")
51print(f"GRPO objective J          : {J:+.4f}")
52print(f"Fraction of rows clipped  : {clipped_frac:.2%}")

Two structural details. First, the order of operations is fixed: rewards → group-normalised advantages → per-completion ratio → clipped surrogate → KL drag → objective. Swap any two of those and the loss is wrong in a way that does not always raise an exception — the run will silently fail to converge. Second, the KL estimator $\rho - \log \rho - 1$ looks unfamiliar but is the GRPO paper's contribution: it is the unique non-negative, unbiased estimator of $\text{KL}[\pi_{\text{old}} \| \pi_{\text{new}}]$ that you can compute from a single sample. Using $\log \rho$ directly (which is unbiased but signed) costs the run its monotone-improvement guarantee.

PyTorch: The Production GRPO Loss

The production version vectorises everything to $(B, G, T)$ — batch of prompts, group of completions, response tokens. Three things become necessary at this scale: a response-token mask (so the prompt doesn't contribute gradient), a reference policy distinct from the old policy (because R1 refreshes $\pi_{\text{old}}$ every step but $\pi_{\text{ref}}$ rarely), and per-token advantage broadcasting.

1import torch
2import torch.nn.functional as F
3
4def grpo_loss(
5    logp_new,   # (B, G, T)  log pi_new(y_t | x, y_{<t})
6    logp_old,   # (B, G, T)  log pi_old(y_t | x, y_{<t})
7    logp_ref,   # (B, G, T)  log pi_ref(y_t | x, y_{<t})
8    rewards,    # (B, G)     scalar reward per completion
9    mask,       # (B, G, T)  1 for response tokens, 0 for prompt / padding
10    *,
11    epsilon: float = 0.2,
12    beta: float = 1e-3,
13):
14    """
15    DeepSeek R1's GRPO loss. No critic, no value head, no GAE.
16    Reference (pi_ref) is the post-SFT model that the policy started from.
17    """
18    # --- Step 1: group-normalised advantages (no critic) -----------------
19    mu = rewards.mean(dim=1, keepdim=True)                  # (B, 1)
20    sd = rewards.std(dim=1, keepdim=True).clamp(min=1e-8)   # (B, 1)
21    A  = (rewards - mu) / sd                                # (B, G)
22    A  = A.unsqueeze(-1)                                    # (B, G, 1) -> broadcast over T
23
24    # --- Step 2: token-level log-ratio and policy ratio ------------------
25    log_ratio = logp_new - logp_old                         # (B, G, T)
26    ratio     = log_ratio.exp()                             # (B, G, T)
27
28    # --- Step 3: PPO clipped surrogate, per token ------------------------
29    unclipped = ratio * A
30    clipped   = ratio.clamp(1 - epsilon, 1 + epsilon) * A
31    surrogate = torch.minimum(unclipped, clipped)           # (B, G, T)
32
33    # --- Step 4: GRPO unbiased KL estimator (vs reference) ---------------
34    # KL = pi_ref/pi_new - log(pi_ref/pi_new) - 1, non-negative, zero at equality
35    log_ref_new = logp_ref - logp_new
36    kl_token    = log_ref_new.exp() - log_ref_new - 1.0     # (B, G, T)
37
38    # --- Step 5: mask + mean over response tokens, then over group/batch -
39    objective = surrogate - beta * kl_token                 # (B, G, T)
40    objective = (objective * mask).sum(dim=-1) / mask.sum(dim=-1).clamp(min=1)
41    return -(objective.mean())                              # minimise -J

Three observations on what makes this code production-ready vs. toy:

1. Two policies, not one. $\pi_{\text{old}}$ is for the importance ratio; $\pi_{\text{ref}}$ is for the KL anchor. R1 keeps them separate because the old policy is refreshed every gradient step (so the ratio stays near 1) while the reference is refreshed every 100–500 steps (so the KL penalty stays meaningful). Conflating them collapses GRPO to standard PPO with a tiny KL coefficient, which loses the entire point.
2. Token-level masking is non-negotiable. The objective is averaged over response tokens only — never the prompt tokens or pad tokens. Skip the mask and the prompt's log-probability contributes to the gradient, which is exactly backward (you would be reinforcing the prompt instead of the response). Most reported "GRPO doesn't work" failures trace to a missing or wrong mask.
3. Group statistics use $\text{std}.\text{clamp}(\min=10^{-8})$. Early in R1 training many prompts return all-zero or all-one reward groups (model is too weak or too strong). Without the clamp the divide is NaN, which propagates everywhere via the optimiser's second moment. The clamp keeps the advantage at zero for those prompts — they contribute no gradient — which is the correct behaviour.

At Massive Scale: Why These Numbers and Not Others

Match the hyperparameters to the bottlenecks they relieve at the 671B-parameter scale R1 was trained on:

Two further constraints become visible only at this scale. First, the rollout phase is sequential per prompt (you cannot generate token 5 before token 4) but parallel across prompts. With $B = 1024$ prompts and $G = 16$ completions each, R1 generates $16384$ sequences in parallel — saturating the inference cluster the rollouts run on. Increasing $G$ beyond 16 would either require more inference GPUs or longer wall-clock per step. Second, the gradient phase is one optimiser step on the entire $16384$-completion batch. Larger effective batches would amortise communication better but run out of GPU memory for the activations; $G = 16$ sits at the upper edge of what fits.

Engineering Reality and Gotchas

Three failure modes earn their flag in every production GRPO log:

1. Reward-zero groups silently kill the gradient. When the model is too weak to solve a prompt, all $G$ completions get reward 0, the group std is zero, the clamp returns $10^{-8}$, and the advantage becomes a vector of zeros — which contributes no gradient. Early in R1 training this happens to ~30% of prompts. The R1 fix is curriculum-style prompt filtering: drop prompts whose recent group-pass-rate is at 0% or 100% from the training mix, because they contribute no learning signal in either case. Without this filter, ~40% of compute is spent on prompts that contribute nothing.
2. The reference policy must be refreshed, but not too often. Refresh $\pi_{\text{ref}}$ every step and GRPO collapses to standard PPO; refresh never and KL grows without bound until $\beta \cdot KL$ dominates the surrogate and learning stalls. R1's reported schedule refreshes every 100–500 steps; the right number depends on how fast per-token KL is growing. A practical rule: refresh when mean per-token KL exceeds $0.5$ (corresponding to $\beta \cdot KL \approx 5 \cdot 10^{-4}$, half the surrogate signal).
3. Length normalisation is the single biggest hidden knob. The factor $1/|y_i|$ in the objective makes long completions contribute the same gradient weight as short ones. This is R1's choice, but Dr.GRPO (Section 15.5) argues it implicitly rewards verbosity and drops it. Switching the normalisation between these two forms — one line of code — changes whether the model learns to think more or less. There is no universally right answer; it depends on what the reward measures and what trade-off you want.

The one sentence to carry forward: GRPO's magic is in the recipe, not the equation — $G = 16, \epsilon = 0.2, \beta = 10^{-3}, \eta = 10^{-6}$ is the single point in hyperparameter space where rule-based rewards on math prompts can drive a 671B model from imitation into emergent reasoning — and the rest of this chapter is about why that point exists, what perturbs it, and how the variants in Section 15.5 move along its frontier.