Rule-Based Rewards for Verifiable Tasks

The Real Problem: Reward Models Lie at Scale

The previous two sections built a neural reward model from human preferences. It works. It also breaks in a very specific, very expensive way: the longer you train against it, the more it lies.

Here is the failure, concretely. You train a 7B reward model on 100k preference pairs. Its validation accuracy is 72%. You plug it into PPO and start optimising the policy against it. For the first few hundred steps everything looks great — reward goes up, eval scores go up. Then around step 500 the curves diverge. Reward keeps climbing. Eval scores plateau, then drop. By step 2000 the reward model is rating the policy's outputs above the human-written gold standard, and the policy is producing fluent-sounding nonsense that hits every pattern the reward model learned to like.

This is reward hacking, and it is the central engineering risk of RLHF at scale. The reward model is a finite neural network. Its decision boundary has cracks. The policy is a much larger neural network whose entire job during RL is to find and exploit those cracks. The bigger and better-trained the policy, the faster it finds them.

The brutal asymmetry: A reward model trained on 100k preference pairs has roughly 100k bits of information about what humans like. A 671B-parameter policy doing 10M rollouts has roughly 10M opportunities per epoch to find a pattern the reward model wrongly labelled as good. Information-theoretically, the policy will always win this race.

Now look at one specific class of tasks — math, code, theorem proving, structured extraction, constrained generation — and notice that we are paying for a reward model we do not need. For these tasks, the human preference is not the ground truth. The ground truth is the ground truth. $x^2 - 5x + 6 = 0$ has solutions 2 and 3 whether or not a labeller agrees. The unit tests pass or they do not. The JSON parses or it does not.

For verifiable tasks, the reward model is a noisy approximation of a cheap, deterministic function. Why train a 7B network to predict what a five-line regex already knows?

Intuition: Use a Judge Made of Code, Not Neurons

Imagine you are training a high-school student. You can grade their algebra homework in two ways. Option A: hire another student to compare answers and tell you which one looks more correct. Option B: solve the problem yourself and check whether the digits match.

Option A is the reward model. It is necessary when the task is subjective — "was this essay persuasive" — but it is wasteful and error-prone when the task is verifiable. Option B is rule-based reward. It is faster, cheaper, more accurate, and impossible to reward-hack in the same way as a neural judge.

Three properties make a task a good candidate for rule-based reward:

1. Deterministic ground truth. The right answer is not a matter of opinion. $2 + 2 = 4$. The unit tests pass. The XML validates.
2. Cheap verification. Checking the answer is orders of magnitude cheaper than generating it. A regex runs in microseconds. A unit test runs in milliseconds. The policy that produced the output ran for seconds and hundreds of GPU-watts.
3. Bounded reward. The verifier outputs a number in a known range (usually $[0, 1]$ or a small integer). This is what makes the group-advantage normalisation in GRPO stable.

When all three hold, you should not be training a reward model. You should be writing 50 lines of Python that the model is graded against millions of times during RL. The Python is the judge.

The Math: A Deterministic, Sparse, Verifier-Defined Reward

Let $\pi_\theta$ be the policy (the language model with parameters $\theta$), $s$ a prompt, and $a \sim \pi_\theta(\cdot \mid s)$ a rollout sampled from the policy. The classical RLHF objective is:

$J(\theta) = \mathbb{E}_{s, a}[\, r_\phi(s, a)\,] - \beta \, \mathrm{KL}\big(\pi_\theta(\cdot \mid s) \,\|\, \pi_{\mathrm{ref}}(\cdot \mid s)\big)$

where $r_\phi$ is the neural reward model with parameters $\phi$, and the KL term anchors the policy to a frozen reference. The whole machinery of the previous section was about learning $r_\phi$ from human preferences.

Rule-based reward replaces $r_\phi$ with an explicit function $R$ that has no learnable parameters. The verifier is the function. Concretely, $R$ is a weighted sum of rule-level scores:

$R(s, a) = \sum_{i=1}^{K} w_i \, r_i(s, a)$

Each $r_i: (s, a) \to [0, 1]$ is a verifier — a piece of code, not a neural net. The weights $w_i \geq 0$ are hand-set hyperparameters. For a math task we might have:

$R_{\mathrm{math}}(s, a) = 1.0 \cdot r_{\mathrm{answer}}(s, a) + 0.1 \cdot r_{\mathrm{format}}(s, a)$

where $r_{\mathrm{answer}}$ is 1 if the extracted boxed value matches the ground truth and 0 otherwise, and $r_{\mathrm{format}}$ is 1 if the rollout has the expected $\langle\mathrm{think}\rangle$/$\langle\mathrm{answer}\rangle$ structure. The maximum possible reward is $R_{\max} = 1.1$.

Substituting back into the RL objective and using GRPO's group-relative advantage:

$A_i = \frac{R(s, a_i) - \mu_G}{\sigma_G + \varepsilon}, \quad \mu_G = \frac{1}{G}\sum_{j=1}^{G} R(s, a_j), \quad \sigma_G^2 = \frac{1}{G}\sum_{j=1}^{G} (R(s, a_j) - \mu_G)^2$

where $G$ is the group size (the number of rollouts sampled per prompt), and $a_i$ is the $i$-th rollout. The mean and standard deviation are computed inside the group, so the advantage $A_i$ has zero mean and unit variance by construction. This is what lets GRPO drop the value network — a critical computational saving when the policy is already 671B parameters.

Finally, the PPO-clipped surrogate is computed token by token:

$L(\theta) = -\mathbb{E}_{s, i, t}\Big[\, \min\big(\rho_{i,t}(\theta) A_i, \,\, \mathrm{clip}(\rho_{i,t}(\theta), 1 - \epsilon, 1 + \epsilon) A_i\big) \Big] + \beta \, \mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}})$

where $\rho_{i,t}(\theta) = \frac{\pi_\theta(a_{i,t} \mid s, a_{i, < t})}{\pi_{\theta_{\mathrm{old}}}(a_{i,t} \mid s, a_{i, < t})}$ is the per-token importance ratio between the current policy and the sampling policy.

Read the surrogate as: for every token in every rollout with positive advantage, push the policy's probability of that token up (clipped so the step is bounded); for every token in every rollout with negative advantage, push it down. The KL penalty stops the whole thing from flying off.

The mathematical novelty of rule-based RL is not in the surrogate — that is just PPO. It is in what fills the slot $R(s, a)$: a deterministic, parameter-free function that returns the same number every time. No noise in the target. No reward-model overfitting. No drift.

The Three Families of Verifiable Tasks

Almost every rule-based reward in production falls into one of three families, distinguished by how the verifier reads the rollout.

Exact-match

Used for any question with a single canonical answer. The verifier extracts the answer with a regex (typically $\boxed{...}$ for math, a tagged span for QA, a number for arithmetic) and compares against the ground truth. The trickiest engineering is the normalisation step: should $0.5$, $1/2$, and $\frac{1}{2}$ all match? The major math benchmarks (MATH, AIME) ship with reference normalisers; using them is the difference between a 60% and a 75% pass@1 measurement on the same model.

Regex / format

Used for structural priors. DeepSeek-R1's breakthrough was showing that even a TINY format reward (weight 0.1) is enough to bootstrap chain-of-thought from a base model that has never seen an SFT chat dataset. The model learns to wrap its reasoning in $\langle\mathrm{think}\rangle$ tags purely because the verifier hands out a small bonus when it does. Format rewards are also how you cheaply enforce JSON-mode, function-calling syntax, and language-consistency (e.g. "every sentence is in English").

Execution

The most expensive verifier. The model emits code; you write the code to disk in a sandbox and run it against hidden unit tests. Reward is the fraction of tests that pass. Two engineering constraints dominate at scale: timeouts (infinite loops are the single most common failure mode) and isolation (the sandbox must prevent the policy's code from corrupting the verifier or eating the network). DeepMind's AlphaCode papers report ~30% of model outputs hit either a timeout or an unhandled exception during early training; the timeout is not a corner case, it's the median rollout.

Manual Numerical Walkthrough

We will compute one GRPO step on a single math prompt with $G = 4$ rollouts. Tiny enough to verify every number; structurally identical to a 671B-parameter run.

Interactive: Score Real Rollouts Against Real Rules

Pick a task family, then pick two rollouts to compare. The widget runs the actual verifier code (regex + exact-match + unit-test accounting) and shows each rule's verdict, the per-rule reward contribution, and the GRPO group-advantage between the two rollouts. Try swapping a correct math rollout for one with a wrong answer but perfect format — watch the answer rule flip from ✓ to ✗ while the format rule stays green, and see the total reward drop from 1.10 to 0.10.

Three things to notice as you play with it:

• The reward is a sum, not a product. A rollout with the right format but wrong answer still gets a small reward, not zero. This is what gives the policy a usable gradient when it knows HOW to look like a reasoner but not yet how to be one.
• Group advantage flips sign at the group mean, not at zero. Two rollouts with rewards 1.10 and 0.10 give advantages +1 and -1, both meaningful. Two rollouts both at 1.10 give advantages 0 and 0 — the gradient is silent because there's nothing to learn from agreement.
• The code verifier is the only fractional one. 5/5 tests passing returns 1.0, 3/5 returns 0.6, 0/5 returns 0.0. This fractional shape is what lets RL on code converge in fewer steps than RL on math, where every signal is binary.

Plain Python: Three Verifiers and an Aggregator

The whole rule-based reward system fits in one file. No model, no GPU, no torch. The verifiers are pure functions; the aggregator is a one-line sum; the GRPO group advantage is five lines of statistics. Read it as a complete training-time component, not a toy.

1"""
2Rule-based rewards for verifiable tasks. Three verifiers, one aggregator,
3one advantage computation. The whole file is pure Python — no GPU, no
4model. The reward signal is computed by code that the human authored once
5and now runs millions of times during RL.
6
7  1. verify_math    — extract \\boxed{...} from a tagged answer and
8                       exact-match against ground truth.
9  2. verify_format  — regex check that the rollout has the
10                       <think>...</think><answer>...</answer> structure
11                       in that order.
12  3. verify_code    — write the model's code to disk, sandbox-run it
13                       against hidden unit tests, count the fraction
14                       that pass.
15
16The aggregator combines verifier outputs into a single scalar reward.
17The advantage step normalises rewards inside a GRPO group so the
18policy gradient is unit-variance.
19"""
20from __future__ import annotations
21import re, subprocess, tempfile, math, statistics
22from dataclasses import dataclass
23from typing import Callable, Sequence
24
25# ─── shared regexes ─────────────────────────────────────────────────────
26TAG_RE   = re.compile(r"<think>([\\s\\S]*?)</think>\\s*<answer>([\\s\\S]*?)</answer>")
27BOXED_RE = re.compile(r"\\\\boxed\\{([^}]+)\\}")
28
29# ─── 1. Math verifier — exact match on \\boxed{...} ────────────────────
30def verify_math(rollout: str, truth: str) -> dict:
31    m = TAG_RE.search(rollout)
32    if not m:
33        return {"format": 0.0, "answer": 0.0, "extracted": None}
34    b = BOXED_RE.search(m.group(2))
35    extracted = b.group(1).strip() if b else None
36    return {
37        "format":  1.0,
38        "answer":  1.0 if extracted == truth.strip() else 0.0,
39        "extracted": extracted,
40    }
41
42# ─── 2. Format verifier — structural reward only ────────────────────────
43def verify_format(rollout: str) -> dict:
44    m = TAG_RE.search(rollout)
45    if not m:
46        return {"structure": 0.0, "think_length": 0.0}
47    return {
48        "structure":   1.0,
49        "think_length": 1.0 if len(m.group(1).strip()) >= 8 else 0.0,
50    }
51
52# ─── 3. Code verifier — sandboxed unit-test execution ───────────────────
53def verify_code(code: str, tests: list[str], timeout: float = 2.0) -> dict:
54    """Run each unit test as 'assert ...'. Return per-test pass/fail."""
55    passes = 0
56    for t in tests:
57        prog = code + "\n" + t + "\n"
58        with tempfile.NamedTemporaryFile("w", suffix=".py", delete=True) as f:
59            f.write(prog); f.flush()
60            try:
61                r = subprocess.run(
62                    ["python", f.name],
63                    capture_output=True, timeout=timeout,
64                )
65                if r.returncode == 0:
66                    passes += 1
67            except subprocess.TimeoutExpired:
68                pass        # infinite loop = fail, not a crash
69    return {"tests_passed": passes, "tests_total": len(tests),
70            "frac": passes / max(1, len(tests))}
71
72# ─── 4. Weighted aggregator ─────────────────────────────────────────────
73@dataclass
74class RuleSpec:
75    name: str
76    weight: float
77    score_fn: Callable[[dict], float]   # rollout-result-dict -> [0, 1]
78
79def aggregate(verifier_out: dict, rules: list[RuleSpec]) -> float:
80    """R(s) = sum_i w_i * r_i(s).  r_i is the score for rule i."""
81    return sum(rule.weight * rule.score_fn(verifier_out) for rule in rules)
82
83# Example rule packs for the three task families
84MATH_RULES   = [
85    RuleSpec("format", 0.1, lambda v: v["format"]),
86    RuleSpec("answer", 1.0, lambda v: v["answer"]),
87]
88FORMAT_RULES = [
89    RuleSpec("structure",    1.0, lambda v: v["structure"]),
90    RuleSpec("think_length", 0.2, lambda v: v["think_length"]),
91]
92CODE_RULES   = [
93    RuleSpec("frac", 1.0, lambda v: v["frac"]),
94]
95
96# ─── 5. GRPO group-advantage — unit-variance baseline ───────────────────
97def group_advantages(rewards: Sequence[float]) -> list[float]:
98    """Group-relative advantage: (R_i - mean(R)) / (std(R) + eps).
99    The whole point of GRPO: drop the critic, use the group mean as the
100    baseline, the group std as the scale. Works because rule rewards are
101    BOUNDED in [0, R_max] so the variance estimate is well-behaved."""
102    if len(rewards) < 2:
103        return [0.0] * len(rewards)
104    mu  = statistics.mean(rewards)
105    sd  = statistics.pstdev(rewards) or 1e-6
106    return [(r - mu) / sd for r in rewards]
107
108# ─── 6. End-to-end on a single prompt ───────────────────────────────────
109def score_group(rollouts: list[str], truth: str) -> dict:
110    """Score G rollouts of the same math prompt. Returns rewards + advantages."""
111    rs = []
112    for r in rollouts:
113        v = verify_math(r, truth)
114        rs.append(aggregate(v, MATH_RULES))
115    return {"rewards": rs, "advantages": group_advantages(rs)}

Read this code with one eye on the structure. The interface between "what counts as right" (the verifier) and "how much it matters" (the rule-spec weight) is what makes the system tunable in production. The R1 team famously bumped the format weight from 0.1 to 0.05 mid-training because the model had already internalised the structural prior — a five-minute config change, no retraining of any neural component.

PyTorch: Plugging Rule Rewards Into a GRPO Loop

Now wire the rule reward into a real training step. The reward function is just one more callable; the rest is standard GRPO — sample, score, advantage, PPO clip, KL penalty. Reading this loop side by side with a standard PPO loop is the easiest way to see exactly which components the rule-based approach lets you delete.

1"""
2GRPO training step with rule-based rewards. Two key takeaways:
3
4  - No reward model. The 'reward function' is just three Python verifiers
5    behind a thin RewardFunction interface. They could be Python regex,
6    a subprocess sandbox, or in production a separate microservice.
7  - No value function. The baseline is the per-prompt reward mean.
8
9We compute the policy-gradient surrogate with PPO clipping (the 'P' part
10of GRPO). Run this in a loop over prompts and you have a complete
11RL-from-rules training pipeline.
12"""
13import torch
14import torch.nn.functional as F
15from dataclasses import dataclass
16from typing import Callable
17
18# ─── 1. The reward function interface ───────────────────────────────────
19@dataclass
20class RewardFn:
21    """A reward function takes (prompt, completion, meta) -> float."""
22    name: str
23    fn: Callable[[str, str, dict], float]
24
25def make_reward(*fns: RewardFn):
26    """Combine multiple reward functions into one scalar."""
27    def combined(prompt: str, completion: str, meta: dict) -> float:
28        return sum(rf.fn(prompt, completion, meta) for rf in fns)
29    return combined
30
31# ─── 2. The GRPO step ───────────────────────────────────────────────────
32def grpo_step(
33    policy:    torch.nn.Module,                       # current policy
34    ref:       torch.nn.Module,                       # frozen reference
35    prompts:   list[str],                             # batch of prompts
36    metas:     list[dict],                            # per-prompt info (truths, tests)
37    reward_fn: Callable[[str, str, dict], float],     # rule-based reward
38    tokenize:  Callable,                              # str  -> tensor
39    detokenize:Callable,                              # tensor -> str
40    G:         int   = 8,                             # group size
41    clip:      float = 0.2,                           # PPO clip
42    kl_coef:   float = 0.04,                          # KL-to-ref penalty
43):
44    """One GRPO update on a batch of prompts."""
45    total_loss = 0.0
46
47    for prompt, meta in zip(prompts, metas):
48        # 2a. Sample G rollouts from the current policy
49        ids = tokenize(prompt).to(policy.device)
50        gens, logp_old = sample_with_logp(policy, ids, G)
51        completions = [detokenize(g) for g in gens]
52
53        # 2b. Score every rollout with the RULE-BASED reward
54        rewards = torch.tensor(
55            [reward_fn(prompt, c, meta) for c in completions],
56            device=policy.device, dtype=torch.float32,
57        )                                              # shape (G,)
58
59        # 2c. Group-relative advantage  (no value network!)
60        adv = (rewards - rewards.mean()) / (rewards.std() + 1e-6)
61
62        # 2d. Re-score under current policy (gradient flows here)
63        logp_new = policy_logprobs(policy, ids, gens)  # shape (G, T)
64        logp_ref = policy_logprobs(ref,    ids, gens)  # shape (G, T)
65
66        # 2e. PPO clipped surrogate, summed over generated tokens
67        ratio = torch.exp(logp_new - logp_old)         # (G, T)
68        surr1 = ratio * adv.unsqueeze(-1)
69        surr2 = torch.clamp(ratio, 1 - clip, 1 + clip) * adv.unsqueeze(-1)
70        pg    = -torch.min(surr1, surr2).mean()
71
72        # 2f. KL-to-reference penalty stops policy collapse on a high-reward mode
73        kl = (logp_new - logp_ref).mean()
74
75        total_loss = total_loss + pg + kl_coef * kl
76
77    total_loss = total_loss / len(prompts)
78    return total_loss
79
80# ─── 3. Wiring a rule-based reward into the loop ────────────────────────
81def math_reward(prompt: str, completion: str, meta: dict) -> float:
82    """Exact-match on \\boxed{...} from a <think>/<answer> formatted rollout."""
83    import re
84    tag = re.search(
85        r"<think>([\\s\\S]*?)</think>\\s*<answer>([\\s\\S]*?)</answer>",
86        completion,
87    )
88    if not tag:
89        return 0.0                                      # no format reward
90    box = re.search(r"\\\\boxed\\{([^}]+)\\}", tag.group(2))
91    if not box:
92        return 0.1                                      # got tags but no boxed
93    return 1.0 + 0.1 if box.group(1).strip() == meta["truth"] else 0.1
94
95reward_fn = make_reward(
96    RewardFn("math", math_reward),
97)
98
99# loss = grpo_step(policy, ref, prompts, metas, reward_fn, tokenize, detokenize)
100# loss.backward(); optim.step()

Two things to count when you look at this loop. The forward-pass models: just two (current policy + frozen reference). Compare to standard PPO-RLHF, which needs four (policy + reference + reward model + value model). At 671B parameters that's ~3 TB of GPU memory you don't have to spend. The trained components: just one — the policy. Compare to PPO-RLHF, which trains two (policy + value net) and treats one as fixed (reward model, trained earlier). The training plumbing collapses.

At Massive Scale: How DeepSeek-R1 Used This

The R1 paper is the canonical example of rule-based rewards pushed to the limit. Phase 1 ("R1-Zero") trained a 671B base model with ZERO supervised fine-tuning data and ZERO human preference data. The entire training signal was two rule rewards:

1. Format reward, weight 0.1: regex match on $\langle\mathrm{think}\rangle ... \langle/\mathrm{think}\rangle\langle\mathrm{answer}\rangle ... \langle/\mathrm{answer}\rangle$ structure, in that exact order.
2. Answer reward, weight 1.0: exact-match on $\boxed{...}$ against ground truth, for math problems; pass-rate on hidden unit tests, for code problems.

That is the whole reward function. The training corpus was ~150k math problems and ~30k code problems. GRPO with group size $G = 16$. Eight thousand training steps. At the end of it, R1-Zero scored 86.7% on AIME 2024 (versus 9.3% for the unmodified base model) — purely from rule-based RL on a model that had never been instruction-tuned.

How does this fit into massive-model training? Three places to watch.

The GPU/CPU split

Policy forward + sampling sits on the GPU cluster (671B model, ~14 H100 nodes worth of inference). Verifier evaluation sits on a CPU worker pool — for math, ~50 cores keep up with the GPU sampler; for code, ~500 cores because each unit-test subprocess is ~100ms. The CPU pool feeds a Redis queue that the GPU-side training loop drains. Drop a verifier worker and the H100s start idling within a few seconds — verifier capacity is a real cluster-planning input, not an afterthought.

The sandbox tax

Sandboxing the code-execution verifier is non-trivial at scale. A single malicious-looking rollout (the model emits $\mathrm{os.system}(\text{"rm -rf /"})$) is harmless if you run it in a per-rollout firejail. It deletes your training cluster if you don't. DeepSeek used a custom microVM (Firecracker- style) per code rollout, with a 2 s wall-clock budget and 256 MB memory cap. Throughput: ~2k code verifications per second per node. Cost: roughly 10% of total training-cluster spend.

Reward determinism gives you reproducibility

Because rule rewards are deterministic, you can re-score any old rollout exactly. This sounds obvious until you compare with RM-based RLHF: the reward model continues to drift as the policy evolves (it gets re-trained periodically on fresh preference data), so a rollout's reward at step 1000 is not comparable to the same rollout's reward at step 5000. With rule rewards, every rollout has a permanent, immutable score. Debugging is suddenly possible.

Engineering Reality: Reward Hacking and the Verifier Tax

Rule-based rewards eliminate one class of reward hacking (gaming a neural RM) and introduce another (gaming a deterministic verifier). The bugs are different. They are mostly cheaper to fix, but you do have to fix them.

Reward hacks we have personally seen, with their fixes

Notice that all six fixes are 1-3 line changes to the verifier. That is the silver lining of rule-based rewards: when reward hacking happens, you can fix it on a coffee break. Compare with neural RM reward hacking, where the fix is "collect another 100k preference pairs and retrain the RM" — measured in days and dollars, not minutes.

The verifier tax

Rule-based rewards trade off in two places that are easy to underestimate.

1. You need ground truth. Every prompt in your training corpus must come paired with a verifiable answer. For math and code this is easy (existing datasets). For domain tasks (medical reasoning, legal extraction, scientific QA), you have to either curate or synthesise the ground-truth set — and if you do it badly, you train the model to be confidently wrong in exactly the ways your dataset is wrong.
2. You need verifier capacity. The CPU/GPU balance is real and unforgiving. A training cluster optimised for pre-training (heavy on GPU, light on CPU) will throughput- starve on rule-based RL because the verifier pool is too small. Plan for ~1 CPU core per concurrent rollout for math, ~10 cores per rollout for code, and provision the cluster accordingly. This is one of the surprise budget items that derails first-time rule-based RL projects.

The take-away for massive-model training is the same as the take-away for any tool in this book: pick the right one for the job. Rule-based rewards are the right tool when ground truth exists, verification is cheap, and the reward range is bounded. For those tasks — and a surprisingly large fraction of the useful work LLMs do is in those tasks — they are faster, cheaper, more interpretable, and more robust than the neural alternative. Use them. The next section adds a second new tool on top: when verification can't be done by regex but CAN be done by another LLM, you reach for generative reward models, the bridge between rule-based and human-preference rewards.