The Bradley-Terry Model for Preferences

The Real Problem: We Cannot Score, Only Compare

Suppose you want to teach a language model what humans consider a good answer. The obvious plan is to ask thousands of labelers to score model outputs on a 1-to-10 scale and train the model to maximize that score. This plan does not work, and it does not work for a deep human reason rather than a technical one.

Humans are excellent comparators and terrible absolute raters. Ask the same person on Monday and Friday to score an essay and you will get two different numbers. Ask two people in the same room and you will get two more. But ask anyone: which of these two essays is better? and the agreement rate jumps from ~50 % (on numerical scoring) to ~75 % (on pairwise choice). The signal is in the comparison, not the scale.

So the data we can actually collect cheaply and reliably is a stream of pairwise outcomes:

• For prompt $x$, the policy produces two candidate responses $y_A$ and $y_B$.
• A human picks one as the winner: $y_A \succ y_B$ or $y_B \succ y_A$.
• We never observe a numeric quality value for either response.

This is the only signal we have. The reward model needs to turn this stream of binary choices into a continuous $r(x, y) \in \mathbb{R}$ — a number that RL can later climb. How? The answer comes from a 1952 paper by Ralph Bradley and Milton Terry on tournament rankings, decades before anyone imagined fine-tuning a transformer.

Intuition: A Hidden Ladder of Quality

Imagine every possible response to a prompt sits on an invisible ladder. The higher a response is on the ladder, the better humans consider it. The exact height is unobservable — we cannot reach in and measure it — but the ladder exists.

When a human is shown two responses, they are not reading each one's ladder height. They are doing something simpler: looking at both, sensing which one is higher, and clicking it. The bigger the height gap, the more confident the click. When two responses are at almost the same height, the click is nearly a coin flip. This is the entire model in one sentence.

The core assumption: the probability that a labeler picks response A over response B is a smooth, monotone function of the gap between their ladder heights. Bigger gap → more confident preference. Equal heights → 50/50.

Two more constraints fall out of the intuition almost for free:

1. Symmetry. $P(A \succ B) + P(B \succ A) = 1$. Someone has to win; ties we will handle separately.
2. Only the gap matters. If we shift every response's ladder height by the same amount, the gaps stay the same and so do all probabilities. The scores are identifiable only up to a global constant.
3. Saturation, not explosion. If A is wildly better than B, the probability should approach 1 but never exceed it. Whatever function we use must map an unbounded gap into $(0, 1)$.

The simplest function that does all three is the sigmoid. That is the Bradley-Terry model.

The Mathematical Idea: Pairwise Probabilities from Scores

Assign each response a latent score $r(y) \in \mathbb{R}$ — its height on the ladder. The Bradley-Terry probability that A beats B is

$P(y_A \succ y_B) = \sigma(r(y_A) - r(y_B)) = \frac{1}{1 + e^{-(r_A - r_B)}}$

Every symbol matters:

• $r(y_A), r(y_B)$ are the latent rewards of the two responses — real numbers, possibly negative, unbounded.
• $r_A - r_B$ is the margin: the gap between heights. A positive margin means A is above B on the ladder.
• $\sigma(z) = 1 / (1 + e^{-z})$ is the logistic sigmoid: monotone, smooth, symmetric around $\sigma(0) = 0.5$, saturating at 0 and 1.

Given a dataset of $N$ preference pairs $\{(y_w^{(i)}, y_l^{(i)})\}_{i=1}^N$ — where $y_w$ is the winner and $y_l$ the loser — the log-likelihood under Bradley-Terry is

$\log \mathcal{L}(r) = \sum_{i=1}^N \log \sigma(r(y_w^{(i)}) - r(y_l^{(i)}))$

We learn the reward function by minimizing the negative of this:

$\mathcal{L}_{\text{BT}}(r) = -\sum_{i=1}^N \log \sigma(r_w^{(i)} - r_l^{(i)})$

This is the reward-model loss at the heart of InstructGPT, Llama 2 Chat, and every RLHF stack since. It is binary cross-entropy in disguise: each pair carries a target label of 1 (“winner truly is the winner”), and we predict the probability of that label using the margin $r_w - r_l$ as the logit.

The gradient with respect to the two scores is beautifully simple. If we write $p = \sigma(r_w - r_l)$, then

$\frac{\partial \mathcal{L}_{\text{BT}}}{\partial r_w} = -(1 - p), \qquad \frac{\partial \mathcal{L}_{\text{BT}}}{\partial r_l} = +(1 - p)$

Two facts to remember from this:

1. The update is the surprise. If $p$ is already close to 1 (the model agrees with the labeler), the gradient vanishes. Confident-correct examples contribute almost nothing. Confident-wrong examples produce the largest updates.
2. The push is symmetric. Winner gets pulled up by $(1 - p)$; loser gets pushed down by the same amount. Sum of gradients is zero — the mean drifts only via the optimizer's weight initialization, not via the data.

Manual Numerical Walkthrough

Numbers anchor the math. Let us run Bradley-Terry by hand on three responses to the same prompt: “Explain photosynthesis to a child.”

Interactive Visualization

Drag the two reward sliders below to see how the margin maps to a preference probability through the sigmoid. The red dot marks your current operating point on the curve. The annotator simulation on the left rolls a biased coin $N$ times — when $P(A) = 0.5$ the outcome is nearly even; when $P(A) = 0.95$ almost every annotator picks A but you can still see the rare upsets that real datasets contain.

Play with the “resample” button — even with the underlying probability fixed, small $N$ gives noisy empirical $\hat{P}$. This is the real obstacle in reward modeling: each preference pair is one Bernoulli trial, and you need either many trials per pair or many distinct pairs to pin down the score field.

Plain Python: Fitting Bradley-Terry by Hand

Before we touch PyTorch, let us write Bradley-Terry from scratch on NumPy. The whole algorithm — including the gradient, the optimizer, and the gauge fix — fits in 30 lines.

1import numpy as np
2
3# Five chess-like "responses" with a hidden true quality.  We will NEVER look
4# at the true score during fitting — only at pairwise win/lose outcomes, the
5# same signal a human RLHF annotator gives an LLM.
6np.random.seed(0)
7n = 5
8true_r = np.array([2.0, 1.0, 0.5, -0.5, -1.5])   # hidden ground truth
9
10def sigmoid(x):
11    return 1.0 / (1.0 + np.exp(-x))
12
13# Build a dataset of pairwise comparisons.  For each pair (i, j) we sample
14# K matches; winner i happens with probability sigmoid(true_r[i] - true_r[j]).
15K = 40
16pairs = []                                       # list of (winner, loser)
17for i in range(n):
18    for j in range(i + 1, n):
19        p = sigmoid(true_r[i] - true_r[j])
20        wins_i = np.random.binomial(K, p)
21        pairs += [(i, j)] * wins_i               # i beats j
22        pairs += [(j, i)] * (K - wins_i)         # j beats i
23
24# Fit Bradley-Terry: learn r in R^n by maximizing log-likelihood of the data.
25# A useful trick — scores are only identifiable up to a constant, so we anchor
26# the mean at zero after every update.
27r = np.zeros(n)
28lr = 0.1
29for step in range(2000):
30    grad = np.zeros(n)
31    loss = 0.0
32    for w, l in pairs:
33        d = r[w] - r[l]
34        p = sigmoid(d)
35        loss -= np.log(p + 1e-12)
36        grad[w] -= (1.0 - p)                     # dL/dr_w = -(1-p)
37        grad[l] += (1.0 - p)                     # dL/dr_l = +(1-p)
38    r -= lr * grad / len(pairs)
39    r -= r.mean()                                # anchor: mean(r) = 0
40
41true_centered = true_r - true_r.mean()
42print("learned:", np.round(r, 3))
43print("truth  :", np.round(true_centered, 3))
44print("corr   :", np.corrcoef(r, true_centered)[0, 1])

Run this and you should see the learned scores match the centered truth within a few hundredths, with correlation above 0.99. The engine that fits a 7-billion-parameter reward model is exactly this loop — just with the lookup-table scores replaced by a transformer forward pass.

PyTorch: The RLHF Reward-Model Loss

Now the production-shaped version. Two changes from the NumPy code:

1. The latent score is produced by a neural network — a transformer with a scalar head — not stored in an array.
2. We write the loss as $\text{softplus}(-(r_w - r_l))$, the numerically stable form of $-\log \sigma$.

(10000, hidden_size)

(hidden_size, 1)

(batch, hidden)

1import torch
2import torch.nn as nn
3import torch.nn.functional as F
4
5# In RLHF, the reward model is a transformer with a scalar head:
6#     reward = w · h_last_token   (a single number per (prompt, response)).
7# At training time each example provides TWO responses to the same prompt:
8#     "chosen" (the human-preferred one) and "rejected".  The Bradley-Terry
9# loss says: P(chosen ≻ rejected) = σ(r_chosen − r_rejected),  and we
10# minimize −log of that probability.
11
12class RewardModel(nn.Module):
13    def __init__(self, hidden_size: int):
14        super().__init__()
15        # In practice this is a frozen-or-fine-tuned LLM trunk followed by
16        # a single linear "value head".  We replace the trunk by an
17        # nn.Embedding to keep the example self-contained and runnable.
18        self.trunk = nn.Embedding(10_000, hidden_size)
19        self.head  = nn.Linear(hidden_size, 1, bias=False)
20
21    def forward(self, token_ids: torch.Tensor) -> torch.Tensor:
22        # token_ids: (batch, seq).   We use the LAST-token hidden state as
23        # the response summary — the standard RLHF convention.
24        h = self.trunk(token_ids)              # (batch, seq, hidden)
25        last = h[:, -1, :]                     # (batch, hidden)
26        return self.head(last).squeeze(-1)     # (batch,)  scalar reward
27
28def bradley_terry_loss(r_chosen: torch.Tensor,
29                       r_rejected: torch.Tensor) -> torch.Tensor:
30    # −log σ(r_chosen − r_rejected)  ==  softplus(−(r_chosen − r_rejected))
31    # softplus is the numerically stable form: it never overflows when the
32    # margin is hugely negative, and it goes to 0 cleanly when the margin
33    # is hugely positive.
34    return F.softplus(-(r_chosen - r_rejected)).mean()
35
36# --- training step on one preference pair ---
37torch.manual_seed(0)
38model = RewardModel(hidden_size=32)
39opt   = torch.optim.AdamW(model.parameters(), lr=1e-3)
40
41# Two fake responses to the same prompt.  In reality these are tokenized
42# strings produced by the SFT policy.
43chosen   = torch.tensor([[5, 12, 88,  3, 41]])
44rejected = torch.tensor([[5, 12, 88, 99, 17]])
45
46for step in range(200):
47    r_c = model(chosen)
48    r_r = model(rejected)
49    loss = bradley_terry_loss(r_c, r_r)
50
51    opt.zero_grad()
52    loss.backward()
53    opt.step()
54
55print(f"r_chosen  = {model(chosen).item():+.3f}")
56print(f"r_rejected= {model(rejected).item():+.3f}")
57print(f"margin    = {(model(chosen) - model(rejected)).item():+.3f}")
58print(f"P(chosen) = {torch.sigmoid(model(chosen)-model(rejected)).item():.3f}")

The shape contract is the entire interface between Bradley-Terry and any transformer:

At Massive Scale: From One LLM Head to a 70B Reward Model

When the toy embedding becomes Llama-3 70B, three things change — but the loss does not.

Compute and memory

Both chosen and rejected are forwarded through the same 70B network. That is two forward passes and two backward passes per example. To save memory, production trainers concatenate chosen and rejected along the batch dimension so the attention mask handles them as one batch of size $2B$. Activation checkpointing and ZeRO-3 sharding are mandatory — a 70B reward model uses the same parallelism recipe as its SFT parent (see Chapter 11 — Tensor Parallelism and Chapter 12 — FSDP / ZeRO).

Data scale

InstructGPT used ~33 000 comparison pairs. Llama 2's reward model used 1.4 million. Anthropic's Helpful-Harmless dataset reports ~170 000 pairs. The data efficiency of Bradley-Terry — recall the toy example recovered 5 scores from 400 noisy comparisons — is what makes these tiny dataset sizes viable for trillion-parameter policies.

Numerical stability

At 70B parameters trained in bf16, the rewards $r_c, r_r$ can drift to magnitudes of 20-40 if unconstrained. Two universal tricks:

• Reward regularization: add a small $\lambda \cdot (r_c^2 + r_r^2)$ term to keep rewards bounded. Llama 2's recipe uses $\lambda = 10^{-3}$.
• Margin loss: if the labeler gave a confidence rating (“strongly prefer” vs “slightly prefer”), require $r_c - r_r \geq m$ by the confidence tier $m$. This is the InstructGPT extension and most modern reward models use some form of it.

Engineering Reality: Ties, Noise, and Reward Hacking

What about ties?

Bradley-Terry as written has no concept of a tie — every comparison must produce a winner. In practice labelers can mark about equal, and two standard fixes exist:

• Drop ties from training. Loses ~5-10 % of data but keeps the loss exact.
• Rao-Kupper extension: introduce a tie probability parameter $\theta > 1$ so that $P(\text{tie})$ grows when the margin is small. Mathematically clean, almost never used in production because dropping ties just works.

Label noise and the 75 % ceiling

When you measure inter-annotator agreement on real RLHF datasets, you get ~75 %. This is the irreducible noise floor — for one in four prompts, two thoughtful labelers genuinely disagree about which output is better. A reward model that scores above 75 % on held-out pairs is not better, it is over-fit. Watch validation pairwise accuracy carefully; the sweet spot for early-stopping is usually 72-74 %.

Why this matters for RLHF and DPO

Bradley-Terry is also the assumption that lets Direct Preference Optimization (DPO) exist. DPO rewrites the BT loss in closed form against a reference policy, skipping the reward-model-then-PPO pipeline entirely. The trick only works because the BT family has a tractable closed-form ratio between the optimal policy and the reference — that derivation is the next section's job.

Reward hacking

Bradley-Terry produces a reward function that is correct on the DISTRIBUTION of responses the SFT policy generates. Push the RL policy too far and it will discover responses outside that distribution where the reward model is wrong — typically responses that look superficially confident, long-winded, or formatted with bullet points and headers, because those features correlate with “chosen” in the training set. This is reward hacking, and the standard defense is the KL penalty against the reference policy that PPO and DPO both bake in. The reward model itself is not the culprit — Bradley-Terry is doing exactly what it was asked. The problem is that the data did not cover what the policy can become.

One sentence to keep: Bradley-Terry takes a stream of binary clicks and turns them into a calibrated scalar reward via the sigmoid of the score gap. Every modern preference-trained LLM is built on this single line of 70-year-old math.