The R1-Zero Hypothesis

The Real Problem: Where Do Reasoning Traces Come From?

For most of the post-ChatGPT era, “teaching” a large language model to reason has meant exactly one thing: collect a corpus of high-quality, step-by-step solutions written by humans (or by another, already-reasoning model), and supervise-fine-tune the base model on that corpus. Every reasoning model from GPT-4's chain-of-thought capability through OpenAI's o1 went through some version of this pipeline: base model → SFT on chain-of-thought → reward model trained on preferences → PPO. Reasoning was treated as a behaviour that had to be taught, and the supervised step was the place where the teaching happened.

That pipeline has one expensive, brittle bottleneck: the SFT corpus. A real reasoning SFT dataset is hundreds of thousands to millions of examples, each containing a problem and a careful, end-to-end worked-out solution. Where do those solutions come from? Three unsatisfying answers: (1) you pay human annotators — expensive, slow, and inconsistent at the scale of millions of examples; (2) you distill from a stronger existing reasoning model — but that only works if someone has already built one, and the resulting model cannot exceed its teacher; (3) you mix the two with rejection sampling — better, but now you need the stronger model AND the human annotators AND a quality filter, and the loop is still bounded by whichever model you started with.

On top of the SFT problem, the conventional pipeline carries two more learned networks: the reward model, which approximates human preference signal and which has its own training cost, scaling problems, and well-documented pathologies (length bias, format bias, reward hacking — see §14.3), and a PPO critic, a second transformer the size of the policy that estimates state-value and which doubles the GPU memory budget of the RL phase. By the time you have all three networks — policy, reward model, critic — plus a frozen reference for the KL term, you are training four networks at once and the SFT step is the cheapest of the four.

Intuition: Latent Skill vs Taught Skill

A modern base model has been trained on a staggering amount of text that already contains reasoning: math textbooks, Stack Overflow threads, GitHub repositories, scientific papers, competition solutions, Wikipedia derivations. By the time the base model has finished pretraining on tens of trillions of tokens, it has seen countless examples of someone working through a problem step by step, catching a mistake, backtracking, and trying again. The patterns are in the weights. The question is whether the policy — the conditional distribution over the next token given a problem statement — will actually use those patterns when prompted.

Here is the analogy that crystallises the R1-Zero hypothesis. Imagine a chess grandmaster who has spent years studying opening theory and endgame manuals but who, in actual games, plays the first move that feels right and then panics when their opponent deviates. The knowledge is in there. They do not need a tutor to teach them how to calculate variations; they need a context in which calculating variations is consistently rewarded. Give them a series of games where playing carefully wins and playing carelessly loses, and over time their behaviour will shift — not because they learned new chess theory, but because the existing theory was now the path of least resistance.

That is what R1-Zero proposes for language models. The base model contains, somewhere in its conditional distribution, a thin slice of probability mass corresponding to long, careful, mistake-correcting chains of thought. The SFT step does not create that slice; it just amplifies it by training the model on examples drawn from it. But amplification by SFT is expensive (you need millions of annotated examples) and limited (you can only amplify what your teachers can produce). Amplification by reinforcement learning against a verifiable reward is, in principle, neither: a regex that checks whether the final answer is correct can be applied to billions of self-generated samples for free.

The R1-Zero Hypothesis, Formally

The R1-Zero paper's central claim can be stated in one sentence with three pieces:

Three load-bearing words in that sentence deserve a closer look. Sufficiently strong means — in practice, in 2024 — a 671-billion-parameter Mixture-of-Experts model (DeepSeek-V3-Base) with ~37B active parameters, trained on 14.8T tokens with a heavy share of math and code. On a 7B model the same recipe substantially underperforms a 7B SFT baseline; on a 671B base model it produces a reasoning model competitive with the strongest closed systems of its era. The hypothesis is not “RL alone works at every scale”; it is “RL alone works at scale.”

Verifiable means the reward function can read the model's output and decide, deterministically and without learning, whether it is correct. Math problems with closed-form answers, coding problems with unit tests, and logic puzzles with a unique solution all admit verifiable rewards. Creative writing, open-ended dialogue, and most chat tasks do not — which is precisely why R1-Zero is a reasoning hypothesis, not a general-purpose RLHF hypothesis. (Chapter 17 covers how R1 stitches together the reasoning-RL phase with conventional alignment to get a model that is good at both.)

No SFT means the policy at step 0 of RL is $\pi_\theta^{(0)} = \pi_{\text{base}}$ — the same model that produces “blue sky 1+1=2 the wave function collapses” when you ask it “What is 17+26?” with no chat template. The reference policy used for the KL penalty is also the base model: $\pi_{\text{ref}} = \pi_{\text{base}}$. There is no SFT checkpoint anywhere in the pipeline. That is the part of the hypothesis that surprised the field most.

Visualizing the Two Pipelines

Toggle between the two paths below. The conventional path has five stages and four trained networks (SFT model, reward model, critic, policy). R1-Zero has three stages and two networks (policy, frozen reference). The arrows do not just represent passes of compute — they represent assumptions about where reasoning comes from.

The Mathematical Idea: GRPO With $\pi_{\text{ref}} = \pi_{\text{base}}$

R1-Zero is, in equation form, the GRPO objective applied with the base model as both the starting policy and the reference. Let $\pi_\theta$ be the policy being trained, with parameters $\theta$ initialised to the base-model weights. For a prompt $x$, the policy samples a group of $K$ completions $\{y^{(1)}, \dots, y^{(K)}\}$. Each completion is scored by the rule-based reward $r(x, y)$, and the group-relative advantage of completion $i$ is

$A^{(i)} = \frac{r(x, y^{(i)}) - \mu}{\sigma + \varepsilon}, \quad \mu = \frac{1}{K} \sum_{j} r(x, y^{(j)}), \quad \sigma = \sqrt{\frac{1}{K} \sum_{j} \bigl(r(x, y^{(j)}) - \mu\bigr)^2}.$

The advantage is a per-completion scalar, positive when the completion was above the group's average and negative when it was below. Every token in completion $i$ inherits the same advantage; GRPO does not allocate credit at the token level. The objective for one prompt is then the standard clipped-PG objective with a KL penalty:

$\mathcal{L}(\theta) = -\, \mathbb{E}_{i, t} \bigl[ \min\bigl( \rho^{(i)}_t \, A^{(i)}, \; \mathrm{clip}(\rho^{(i)}_t, 1-\epsilon, 1+\epsilon) \, A^{(i)} \bigr) \bigr] + \beta \cdot \mathbb{E}_{i, t} \bigl[ \mathrm{KL}\bigl( \pi_\theta \,\Vert\, \pi_{\text{ref}} \bigr) \bigr],$

where $\rho^{(i)}_t = \pi_\theta(y^{(i)}_t \mid x, y^{(i)}_{<t}) / \pi_{\theta_{\text{old}}}(y^{(i)}_t \mid x, y^{(i)}_{<t})$ is the importance ratio between the current policy and the policy snapshot at the start of the GRPO step, and $\epsilon$ is the PPO-style clipping range. The KL term penalises drift away from the reference policy $\pi_{\text{ref}}$, and the coefficient $\beta$ controls how willing the policy is to trade reward for distributional drift.

Every symbol in those two equations is identical to a standard PPO + KL-to-SFT setup — except for two pieces. First, $A^{(i)}$ is the group-relative advantage rather than $r^{(i)} - V_\phi(s^{(i)})$, eliminating the critic network $V_\phi$. Second, $\pi_{\text{ref}}$ is the base model, not the SFT model. The first change cuts memory in half; the second change is the whole hypothesis.

Rule-Based Reward: No Critic, No Reward Model

The reward function in R1-Zero has two scalar components and is defined entirely in code — not learned, not human-rated, not approximated. For a math problem with gold answer $y^\star$, the reward of a completion $y$ is

$r(x, y) \;=\; \underbrace{\mathbb{1}\bigl[\, \mathrm{extract}(y) = y^\star \,\bigr]}_{\text{accuracy}} \;+\; \underbrace{0.1 \cdot \mathbb{1}\bigl[\, \mathrm{tagged}(y) \,\bigr]}_{\text{format}},$

where $\mathrm{extract}(y)$ pulls the contents of the <answer> tag out of the completion and $\mathrm{tagged}(y)$ checks whether the “think then answer” template is present. The accuracy reward is binary and dominant (weight 1); the format reward is also binary but small (weight 0.1). Both are deterministic functions of the text and the gold answer.

Three observations follow immediately. First, this reward function runs in microseconds per completion — comparable to the tokenizer, orders of magnitude cheaper than a learned reward model. Second, there is no gradient signal flowing into the reward function, which means there is no reward hacking in the usual sense (you cannot “adversarially exploit” a regex by writing nicer prose). Third, the reward is sparse but verifiable: most completions in early training get 0.0 or 0.1, and only the ones that actually solve the problem get 1.1. The model has to find a correct solution by sampling before it can learn anything — which is why the experiment only works at scale, where the base model already gets a non-negligible pass@K on the training set.

Manual Numerical Walkthrough: One GRPO Step on a Math Problem

Open the panel below and follow the arithmetic with a pencil. The numbers are tiny on purpose — once you can compute one GRPO step by hand, the PyTorch implementation is just shape-bookkeeping on top of this.

Interactive: Group-Relative Advantages

Drag the reward sliders to feel how the advantage signal responds. Two preset edge cases are worth visiting. (1) The easy problem: all four rewards at 1.1. The group std collapses to zero, every advantage becomes 0/0 (rescued by the $\varepsilon$), and the model learns nothing. That is correct behaviour — if every sample succeeds, there is nothing to amplify. (2) The hard problem: all four rewards at 0. Same outcome, zero advantages, no learning signal. For R1-Zero to make progress, the base model must already produce a mixed group: some right, some wrong, on the same prompt. This is why R1-Zero only works on a base model strong enough to get non-trivial pass@K on the training set. That is the single biggest scale-dependent condition for the hypothesis to hold.

Plain Python: Rule-Based Reward + Group Advantage

Here is the entire R1-Zero learning signal — reward function plus group-relative advantage plus a sketch of what the policy update does to log-probabilities — in 60 lines of plain Python. The transformer is replaced by a list of strings; the gradient step is replaced by a manual log-probability update; but the math is identical to what runs on 671B parameters.

On a real DeepSeek-V3-Base, problem 'What is 17+26?' with temperature=1.0 and K=16 produces roughly 4 correct + format-OK, 6 correct but un-formatted, 3 wrong answers in the format, 3 incoherent. Base-model pass@1 ≈ 25%; pass@64 ≈ 70%.

PATTERN.search('<think>...</think><answer>43</answer>').group(2) == '43'.  Compared char-for-char with gold='43'.  No tokenization. No model. ~10 microseconds per completion.

reward('<think>17+26=43</think><answer>43</answer>', '43') = (1.1, 1.0, 0.1)
reward('The answer is 43.', '43')                          = (0.0, 0.0, 0.0)
reward('<think>17+26=33</think><answer>33</answer>', '43') = (0.1, 0.0, 0.1)

totals = [1.1, 0.0, 0.1, 0.0]

mu = (1.1 + 0.0 + 0.1 + 0.0) / 4 = 0.30. This is the 'expected reward at this prompt' under the current policy.

pstdev([1.1, 0.0, 0.1, 0.0]) ≈ 0.464.  sd ≈ 0.464 + 1e-8 ≈ 0.464.

advantages = [+1.724, -0.647, -0.431, -0.647]. The correct + formatted completion has the largest positive advantage by far.

Completion 0 (correct + formatted) has lp = -3.2 — already moderately likely under the base model. Completion 3 (garbled) has lp = -7.5 — already very unlikely.

new_lp[0] = -3.2 + 0.172 = -3.028 (completion 0 is now MORE likely)
new_lp[2] = -2.8 - 0.043 = -2.843 (completion 2 is now LESS likely)

1"""
2The two computations at the heart of R1-Zero, in 60 lines of pure Python.
3
4R1-Zero replaces the *two* hardest pieces of conventional RLHF:
5  - the SFT-on-CoT step       -> nothing (we start from the base model)
6  - the learned reward model  -> a rule-based scorer
7  - the PPO critic            -> a group-mean baseline (GRPO)
8
9Everything below runs on plain integers and floats. No transformer, no
10autograd, no GPU. Once you can read this, the production code is just
11shape-bookkeeping on top of it.
12"""
13
14import math
15import re
16from statistics import mean, pstdev
17
18# ---------------------------------------------------------------------------
19# 1. A single math problem and four candidate completions sampled from
20#    the *base* model (no SFT). One is right + formatted, one is right
21#    but unformatted, one is formatted but wrong, one is garbled. Real
22#    R1-Zero samples K=16-64 per problem; we use K=4 so the arithmetic
23#    fits on the page.
24# ---------------------------------------------------------------------------
25
26problem = "What is 17 + 26?"
27gold    = "43"
28
29completions = [
30    # Correct content, correct format.
31    "<think>17 + 26 = 17 + 20 + 6 = 37 + 6 = 43.</think><answer>43</answer>",
32    # Correct content, missing the <think> tag (format penalty).
33    "The answer is 43.",
34    # Wrong arithmetic, correct format.
35    "<think>17 + 26 = 33.</think><answer>33</answer>",
36    # Total nonsense — base model wandered off (no SFT to anchor it).
37    "</think> blue sky 1+1=2 the wave function collapses 12345",
38]
39
40# ---------------------------------------------------------------------------
41# 2. Rule-based reward. TWO components only:
42#       r_format   in {0, 0.1}    -> 0.1 iff <think>...</think><answer>...</answer> shape is present
43#       r_accuracy in {0, 1}      -> 1   iff the extracted answer equals the gold answer
44#    Total reward r = r_accuracy + r_format. No reward model, no human grader.
45# ---------------------------------------------------------------------------
46
47PATTERN = re.compile(r"<think>(.*?)</think>\s*<answer>(.*?)</answer>", re.DOTALL)
48
49def reward(completion: str, gold: str) -> tuple[float, float, float]:
50    m = PATTERN.search(completion)
51    if m is None:
52        return 0.0, 0.0, 0.0
53    r_format   = 0.1
54    extracted  = m.group(2).strip()
55    r_accuracy = 1.0 if extracted == gold else 0.0
56    return r_accuracy + r_format, r_accuracy, r_format
57
58rewards = [reward(c, gold) for c in completions]
59
60print("idx |  total  acc  fmt  | text (truncated)")
61for i, ((tot, acc, fmt), c) in enumerate(zip(rewards, completions)):
62    short = (c[:50] + "...") if len(c) > 50 else c
63    print(f" {i}  | {tot:5.2f} {acc:4.1f} {fmt:4.2f}  | {short}")
64
65# ---------------------------------------------------------------------------
66# 3. Group-relative advantage. GRPO replaces the PPO critic V(s) with the
67#    sample-mean reward of the K completions for THIS prompt, and then
68#    z-normalises so the policy update is scale-invariant across problems.
69#       A_i = (r_i - mean(r)) / (std(r) + eps)
70# ---------------------------------------------------------------------------
71
72totals = [t for t, _, _ in rewards]
73mu     = mean(totals)
74sd     = pstdev(totals) + 1e-8
75advantages = [(r - mu) / sd for r in totals]
76
77print(f"\nmean reward across group: {mu:.3f}")
78print(f"std  reward across group: {sd:.3f}")
79print("idx |  reward   advantage  | sign")
80for i, (r, a) in enumerate(zip(totals, advantages)):
81    sign = "+ (reinforce)" if a > 0 else "- (suppress) "
82    print(f" {i}  | {r:5.2f}   {a:+6.3f}    | {sign}")
83
84# ---------------------------------------------------------------------------
85# 4. The R1-Zero update direction. For each token in completion i, the
86#    policy gradient is roughly:
87#       grad_theta log pi(token | context) * A_i
88#    Positive A_i pulls the token's log-prob UP, negative A_i pushes it
89#    DOWN. KL to a reference model keeps us close to a sensible prior;
90#    in R1-Zero pi_ref = pi_base, so the KL term penalises drift away
91#    from the *starting* model. No SFT checkpoint involved anywhere.
92# ---------------------------------------------------------------------------
93
94# Toy: pretend the per-token log-prob of completion i is sum(log pi) = lp_i.
95# An update with learning rate eta moves lp_i by eta * A_i (sign-only here).
96lp = [-3.2, -1.1, -2.8, -7.5]   # base-model log-prob of each completion
97eta = 0.1
98new_lp = [lp_i + eta * a for lp_i, a in zip(lp, advantages)]
99
100print("\nstep 0:    lp =", [round(x, 3) for x in lp])
101print("after 1: new_lp =", [round(x, 3) for x in new_lp])
102print("\nCompletion 0 (correct + formatted) moved UP; completions 2 and 3 (wrong) moved DOWN.")
103print("That is the R1-Zero learning signal in its entirety.")

PyTorch: The R1-Zero GRPO Step

Stepping up from plain Python to PyTorch changes very little conceptually. The reward function stays a regex. The advantage stays a one-liner. The policy is now a HuggingFace causal LM, the reference is a frozen copy of the base model, and the log-prob math is done in tensors instead of Python floats. The hardest part of production GRPO is not in this file — it is in the parallel sampling infrastructure (vLLM-based rollout servers, async reward scoring, KL distillation across stages). The inner loop itself is simpler than PPO + RLHF, not more complex.

Memory budget for 7B R1-Zero training: 7B policy (training) + 7B ref (frozen, no optimizer) ≈ 14B params. For 7B PPO-RLHF: 7B policy + 7B critic + 7B reward + 7B ref = 28B. Half the GPU memory before you even count activations.

rule_based_reward('<think>17+26=43</think><answer>43</answer>', '43') = 1.1
rule_based_reward('The answer is 43.', '43')                          = 0.0

On A100s, generate() with K=16 and max_new=1024 takes ~6 seconds per problem. With K=64 it takes ~24 seconds — sampling dominates the wall-clock cost of GRPO, not the gradient step.

rewards.shape = (16,)  with values like tensor([1.1, 0.0, 1.1, 0.1, 0.0, 1.1, 0.0, 0.0, ...]) — typically a sparse but positive signal once training has been running for a few hundred steps.

rewards = [1.1, 0.0, 0.1, 0.0]  =>  mean=0.3, std≈0.464  =>  A = [+1.72, -0.65, -0.43, -0.65].

ref is loaded once at startup with .eval() + .requires_grad_(False). Its memory cost is just the parameters (no optimizer state), so it adds ~14GB for a 7B-bf16 model. No backward pass touches it.

logp_pi.shape = logp_ref.shape = (K, T-1). For K=16, T=1024 that is 16,368 scalars per group — the entire signal that drives the gradient step.

If prompt is 50 tokens and completion is 800 tokens, comp_mask sums to 800 per row. For K=16 that is 12,800 contributing positions per training step.

Typical numbers at step 1000: loss ≈ 0.45, PG component ≈ -0.62 (negative because positive advantage means lower loss), KL ≈ 0.04 × 26 ≈ 1.04. KL term dominates early; PG term dominates late.

For a 7B policy: 14GB params + 14GB grads + 56GB AdamW state ≈ 84GB. For a 7B reference: 14GB params only. Ref is essentially free.

Step 0: loss=0.69, reward=0.15, pass@K=0.4, format=0.6 (base model gets format right sometimes, accuracy rarely).
Step 5000: loss=0.12, reward=0.88, pass@K=0.97, format=0.05 (model is now usually right; few completions are formatted-but-wrong).

1"""
2A minimal R1-Zero training step on top of a HuggingFace causal LM.
3
4This is what runs inside the inner loop of the DeepSeek-R1-Zero codebase
5(and TRL.GRPOTrainer, and OpenRLHF.GRPO) once the parallelism, sampling
6servers, and checkpointing are stripped away. The point of this file is
7to show that the *learning signal* is exactly the plain-Python version
8above, just operating on tensors of shape (K, T) instead of lists of K
9strings.
10
11Two contracts to remember:
12
13  1. pi_ref IS pi_base. We never load an SFT checkpoint. The reference
14     policy for the KL penalty is a frozen copy of the same base model
15     we are training. Drift is measured *against the base model itself*.
16
17  2. The reward is a Python function, not a neural network. The reward
18     model in conventional RLHF gets replaced by a regex + a string
19     comparison. No backward pass through the scorer.
20"""
21
22import re
23import torch
24import torch.nn as nn
25import torch.nn.functional as F
26from transformers import AutoModelForCausalLM, AutoTokenizer
27
28PATTERN = re.compile(r"<think>(.*?)</think>\s*<answer>(.*?)</answer>", re.DOTALL)
29
30
31def rule_based_reward(text: str, gold: str) -> float:
32    """The entire 'reward model' for R1-Zero on a math problem."""
33    m = PATTERN.search(text)
34    if m is None:
35        return 0.0
36    extracted = m.group(2).strip()
37    return (1.0 if extracted == gold else 0.0) + 0.1
38
39
40@torch.no_grad()
41def sample_group(policy, tokenizer, prompt: str, K: int = 16, max_new: int = 1024):
42    """Sample K completions from the policy at temperature 1.0."""
43    enc = tokenizer(prompt, return_tensors="pt").to(policy.device)
44    out = policy.generate(
45        enc["input_ids"].expand(K, -1),
46        max_new_tokens=max_new,
47        do_sample=True, temperature=1.0, top_p=1.0,
48        return_dict_in_generate=True,
49    )
50    texts = tokenizer.batch_decode(out.sequences[:, enc["input_ids"].size(1):],
51                                   skip_special_tokens=True)
52    return out.sequences, texts  # (K, T_total)  +  list[str]
53
54
55def grpo_step(
56    policy: nn.Module,         # the model we are training (init: base)
57    ref:    nn.Module,         # frozen copy of the *base* model
58    tokenizer,
59    prompt: str,
60    gold: str,
61    optimizer,
62    K: int = 16,
63    beta: float = 0.04,        # KL coefficient (DeepSeek-R1-Zero used 0.001-0.04)
64    clip_eps: float = 0.2,
65):
66    # 1. Sample K completions from the current policy.
67    seqs, texts = sample_group(policy, tokenizer, prompt, K=K)
68
69    # 2. Score every completion with the rule-based reward.
70    rewards = torch.tensor(
71        [rule_based_reward(t, gold) for t in texts],
72        device=policy.device, dtype=torch.float32,
73    )                                              # (K,)
74
75    # 3. Group-relative advantage. NO critic network involved.
76    A = (rewards - rewards.mean()) / (rewards.std() + 1e-8)   # (K,)
77
78    # 4. Forward pass through the *policy* and the *reference* model
79    #    on the FULL sequence (prompt + completion). We need per-token
80    #    log-probs to compute both the policy-gradient term and the KL.
81    attention_mask = (seqs != tokenizer.pad_token_id).long()
82    out_pi  = policy(input_ids=seqs, attention_mask=attention_mask)
83    out_ref = ref   (input_ids=seqs, attention_mask=attention_mask)
84
85    # Per-token log p_theta(y_t | y_<t) and p_ref(y_t | y_<t).
86    logits_pi  = out_pi.logits[:,  :-1, :]
87    logits_ref = out_ref.logits[:, :-1, :]
88    targets    = seqs[:, 1:]
89    logp_pi    = F.log_softmax(logits_pi,  dim=-1).gather(-1, targets.unsqueeze(-1)).squeeze(-1)
90    logp_ref   = F.log_softmax(logits_ref, dim=-1).gather(-1, targets.unsqueeze(-1)).squeeze(-1)
91
92    # 5. Only the COMPLETION tokens contribute to the loss; mask out the prompt.
93    prompt_len = tokenizer(prompt, return_tensors="pt").input_ids.size(1)
94    comp_mask  = torch.zeros_like(logp_pi)
95    comp_mask[:, prompt_len - 1 :] = 1.0
96    comp_mask  = comp_mask * attention_mask[:, 1:]
97
98    # 6. The R1-Zero / GRPO loss.
99    #    - Policy-gradient term: -A_i * sum_t log_pi(token)        (per completion)
100    #    - KL penalty:           +beta * sum_t (log_pi - log_ref)  (per token)
101    pg_per_token = -A.unsqueeze(-1) * logp_pi                       # (K, T-1)
102    kl_per_token = logp_pi - logp_ref                               # (K, T-1)
103    loss = ((pg_per_token + beta * kl_per_token) * comp_mask).sum() / comp_mask.sum()
104
105    # 7. Backprop and step. The reference model is frozen — no autograd
106    #    flows into it, so no double-memory cost beyond the forward pass.
107    optimizer.zero_grad()
108    loss.backward()
109    torch.nn.utils.clip_grad_norm_(policy.parameters(), max_norm=1.0)
110    optimizer.step()
111
112    return {
113        "loss":     loss.item(),
114        "reward":   rewards.mean().item(),
115        "pass@K":   (rewards > 0.5).any().item(),  # any completion scored a correct answer?
116        "format":   ((rewards > 0.0) & (rewards < 0.5)).float().mean().item(),
117    }

At Massive Scale: Why R1-Zero Worked on 671B and Not on 7B

The R1-Zero hypothesis is sensitive to scale in a way that almost no other RL technique is. Three concrete reasons explain why DeepSeek published a 671B success story and not a 7B one, and why small-model replications of R1-Zero have been so much weaker than their large-model siblings.

Pass@K floor of the base model

R1-Zero can only learn from a group of $K$ completions if at least one of them gets a non-zero reward. If every completion fails, the group mean is 0, every advantage is 0, and the gradient is 0 — the model has gathered no information from the prompt. The relevant statistic is therefore the base model's $\text{pass@}K$ on the training set:

$P(\text{group has signal}) = 1 - \bigl(1 - p_1\bigr)^K,$

where $p_1$ is the base model's single-shot accuracy on a problem. For DeepSeek-V3-Base on AIME-level math, $p_1 \approx 0.15$; with $K = 16$ the probability that the group contains at least one correct completion is $1 - 0.85^{16} \approx 0.93$ — almost every prompt produces useful signal. For a 7B base model with $p_1 \approx 0.02$, the same calculation gives $1 - 0.98^{16} \approx 0.28$ — less than a third of prompts produce signal, and training is starved.

Memory budget and the death of the critic

At 671B parameters in bf16, the policy alone is ~1.34 TB of weights. Add gradients (1.34 TB) and AdamW optimizer state (5.4 TB) and you are at ~8 TB for the policy without activations — already requiring hundreds of H800s. PPO would demand a critic of the same size, doubling the budget. GRPO's replacement of the critic with a one-line group-mean baseline is what makes R1-Zero feasible at this scale, not just elegant. At 7B the saving is real but not decisive; at 671B it is the difference between a possible experiment and an impossible one.

Sampling throughput and the rollout cost

Each GRPO step samples K completions per prompt at up to 32K tokens each. With K=64 and a 64-prompt batch, one step generates ~130M tokens of completion — comparable to a meaningful slice of the pretraining corpus, every step. R1-Zero was only feasible because DeepSeek had built a fully separated rollout architecture: a fleet of vLLM inference servers running the policy snapshot, decoupled from the training cluster running the gradient step, with the policy weights synced periodically across them. Without that infrastructure, the wall-clock cost of a single GRPO step on a 671B model would be measured in hours rather than minutes.

Engineering Reality: What Pure RL Breaks

R1-Zero is a strong reasoning model and a weak chat model. The same minimalism that makes the recipe work also produces a set of failure modes that you do not see in SFT-trained systems. The chapter's remaining sections (§16.3–§16.6) go into each in detail; here is the short catalogue so the reader knows what is coming.

• Language mixing. With no readability reward and a base model that has seen Chinese, English, and code in similar proportions, the policy can drift toward thinking in one language and answering in another. Inside a single <think> block, R1-Zero will sometimes switch from English prose to Chinese characters to Python syntax and back, all in the service of solving the problem. The answer is still correct; the trace is unreadable to a human reviewer.
• The “aha moment.” Around steps 1k–3k of training, R1-Zero begins spontaneously emitting self-correction phrases — “Wait, that's wrong, let me try a different approach” — without ever having been shown such phrases in an SFT corpus. The next section traces this phenomenon and its implications. It is the single most-cited result of the R1-Zero paper.
• Reward hacking is constrained but not eliminated. Because the reward is a deterministic regex, you cannot “fool the reward model” the way you can with a learned scorer. But the regex can be gamed: the policy can learn to emit <answer>43</answer> inside the <think> block to maximise the chance the regex extractor picks the right string. The DeepSeek paper documents several such micro-exploits and the regex hardening required to close them.
• No chat capability. R1-Zero is trained exclusively on verifiable-reward problems. On open-ended user requests it continues to output <think>…</think><answer>…</answer> even when the user wanted a casual answer. R1 (the production model) fixes this with two further training stages on top of R1-Zero — covered in chapter 17.
• Sensitivity to reward-function bugs. Because there is no learned reward model to smooth over edge cases, a bug in the regex propagates directly into the policy. An early version of R1-Zero's extractor accepted any number inside the <answer> tag, including trailing whitespace and punctuation. The policy learned to emit “43.” (with a period) on problems where the gold was “43” — harmless until the extractor was tightened, at which point pass-rate collapsed overnight. The reward function in pure RL is, in effect, part of the model architecture.

The single sentence to remember. R1-Zero's bet was that reasoning is latent in the base model and that a regex, applied to enough samples, is a sharper teacher than a million human annotators. The bet won at 671B; whether it generalises beyond verifiable-reward domains is the open question of the next several years of post-training research.