Teaching a Model What We Prefer

There is no loss function for “good”

Pretraining works because the label is already in the data. You take a corpus, slide a window across it, and at every position the thing you want the model to predict is sitting right there — it is the next token. The loss is cross-entropy, it is defined per position, it is differentiable in the logits, and the whole apparatus of backpropagation runs through it without complaint. A trillion tokens of supervision, free, requiring no annotator.

Now try to write down the loss for the property we actually want out of a deployed assistant. Not next-token accuracy but helpfulness. Or harmlessness, or honesty, or the vastly under-specified quality of “this is the answer a thoughtful expert would have given.” None of these is a function of the next token. They are properties of a whole completion, judged in context, by a person, against standards that person cannot fully articulate.

This is not a small gap. It is a structural one, and it breaks the chain rule in two separate places.

First, there is no derivative. Backpropagation needs \(\partial \mathcal{L}/\partial \theta = (\partial \mathcal{L}/\partial y)(\partial y/\partial \theta)\). The right factor is fine — that is just the model. The left factor requires the loss to be a differentiable function of the model's output. A human reading a paragraph and saying “this one” is not a differentiable function of anything. You cannot perturb the paragraph by \(\epsilon\) and ask the human for the resulting change in judgement.

Second, there is no throughput. Even if the judge were differentiable, they are slow. A pretraining run consumes something on the order of \(10^{13}\) supervised targets. A well-funded preference-collection effort produces perhaps \(10^5\) to \(10^6\) comparisons. That is eight orders of magnitude. Whatever we build has to make each human judgement go extraordinarily far.

Everything that follows in this article is a response to those two facts. There are exactly three families of response, and it is worth naming them now because the rest of the field is variations on them.

1. Imitate the judge. Ask people to write the answer they would have preferred, then train on it with the same cross-entropy loss you already have. This is supervised fine-tuning, and it is cheap in machinery and expensive in people.
2. Model the judge. Train a differentiable stand-in — a reward model — that predicts what the human would have said, then optimise the policy against the stand-in with reinforcement learning. This is classical RLHF.
3. Skip the stand-in. Show that under the objective in (2), the policy and the reward are two encodings of the same object, so you can fit the policy directly to the comparisons without ever instantiating a reward model. This is DPO, and it is a piece of algebra rather than a new idea about preferences.

Route 1 has a ceiling that is easy to state and easy to underestimate. Cross-entropy on a demonstration only ever pushes probability up on the one sequence the demonstrator happened to write. It contains no information about what was bad, and for open-ended generation there are enormous numbers of acceptable answers, so training the model to put mass on exactly one of them is a strange thing to ask. Preference data is the first signal in the whole pipeline that carries negative information: not just this is good, but this one rather than that one.

Route 2 is where the field actually started, and understanding why requires going back before language models entirely.

Where preference learning came from

The idea that you can recover a reward function from comparisons rather than from a specification predates large language models by a long way. It sits in a lineage that runs through inverse reinforcement learning, preference-based policy learning, and interactive shaping, and the version that mattered for us arrived in 2017.

Why the field moved this way. Directly specifying “good behavior” was impossible, but choosing the better of two examples was easy. A reward model made those sparse choices cheap to reuse; RL then searched beyond the labeled examples. Moving to language added a new danger—the optimizer could trade away fluency while chasing reward—so the reference-policy KL became the leash. Scaling this loop produced the standard RLHF pipeline, and its cost and fragility later motivated direct objectives such as DPO.

Notice that the pipeline has a peculiar shape: two of its five stages exist only to manufacture a differentiable signal. Stage 3 converts human judgement into a dataset; stage 4 converts that dataset into a function with a gradient. Everything after this section is about whether those two stages are necessary, what they cost when they go wrong, and — in the case of DPO — whether they can be algebraically eliminated.

From comparisons to a scalar field

We want a differentiable function \(r_\phi(x,y) \in \mathbb{R}\) that scores completions. The data we can afford to collect is comparisons. The bridge between the two is a hundred-year-old piece of psychometrics.

Why ask for a ranking rather than a rating

Suppose we asked annotators for absolute scores instead — rate this answer out of ten. Three things go wrong immediately. Annotators anchor: the first few examples they see set their internal scale, and different annotators end up on different scales. Scales drift within a session as the annotator recalibrates. And the mapping from “quality” to “a number out of ten” is a second, unrelated cognitive task that we are asking a tired person to perform reliably thousands of times.

Comparisons dodge all three. “B is better than A” requires no shared scale, no memory of previous items, and no numeric translation. It is also the judgement humans are demonstrably best at: relative discrimination is easier than absolute magnitude estimation in essentially every perceptual and evaluative domain that has been studied. The cost is that a comparison carries less information than a rating — one bit rather than three or four — but bits from a reliable channel beat bits from a noisy one.

The latent-utility model

Here is the modelling assumption, stated plainly. We posit that there exists a scalar \(r^\star(x,y)\) — the true quality — and that an annotator's realised judgement of a response is that quality plus noise:

The annotator picks whichever response has the larger realised utility. Different noise distributions give different link functions, and the choice is not innocent:

Had we assumed Gaussian noise instead we would have got Thurstone's model and a probit link, \(\Phi(\Delta/\sqrt{2})\). In practice logistic and probit are nearly indistinguishable after rescaling, which is a relief: the choice of noise family is doing less work than the assumption that a single latent scalar exists at all.

There is a second, purely algebraic way to read the same equation. Write \(\Pr[y_w] = e^{r_w}/(e^{r_w} + e^{r_l})\) — a two-item softmax over exponentiated scores. Bradley–Terry is a softmax, which is why the extension to a full ranking over \(K\) items (the Plackett–Luce model) is exactly a sequence of softmaxes over shrinking candidate sets, and why the DPO paper can handle ranked data with almost no extra work.

The loss

Given a dataset \(\mathcal{D} = \{(x, y_w, y_l)\}\), fit \(r_\phi\) by maximum likelihood. Negative log-likelihood of the Bradley–Terry model is:

That is binary logistic regression, where the “feature” is a difference of network outputs. Three properties of this loss will come back to haunt or help us, so it is worth pinning them down now.

Shift invariance. For any function \(c(x)\), the rewards \(r_\phi\) and \(r_\phi + c(x)\) produce identical likelihood, because only differences within a prompt ever appear. The reward is therefore identified only up to an arbitrary per-prompt constant. Two consequences follow. Comparing reward values across prompts is meaningless. And — this is the hinge on which the entire DPO derivation turns — any per-prompt additive term in a candidate reward function is free to be discarded.

Scale and noise are the same knob. If the noise has scale \(s\) rather than 1, then \(\Pr = \sigma(\Delta/s)\). Doubling all rewards is indistinguishable from halving the annotator noise. This means the numeric scale of a fitted reward model has no intrinsic meaning either; it is fixed only by the amount of disagreement in the training labels. When you later choose \(\beta\) in the KL objective, you are choosing it relative to a reward scale that was itself set by annotator noise. That is one reason \(\beta\) does not transfer between datasets.

The gradient anneals itself. Differentiate:

The magnitude of the update is \(\sigma(-\Delta)\): near 1 when the model currently gets the pair wrong, near 0 when it already gets it confidently right. The loss builds its own curriculum, spending gradient only where it is currently mistaken. Hold onto that shape. It reappears, in exactly the same functional form, as the weighting term in the DPO gradient.

What Bradley–Terry quietly assumes

The model buys us a differentiable loss, and the price is a set of assumptions that are worth stating out loud because most of the field's later troubles are traceable to one of them.

A single scalar orders everything. Bradley–Terry imposes a total order on completions. Preferences that cycle — the annotator pool prefers A to B, B to C, and C to A — cannot be represented by any assignment of scalars, and population-level intransitivity is a documented empirical phenomenon in human choice, not a hypothetical. It also shows up in LLM-generated preference annotations. This is the starting point for the Nash-learning line of work that treats alignment as finding an equilibrium of a preference game rather than the maximum of a preference function.

All annotators share one \(r^\star\). Preferences are heterogeneous. Fitting one Bradley–Terry model to a pooled dataset returns something like a population average, which may be a response nobody in particular wanted.

Noise is independent of the items. The derivation assumed \(\varepsilon_y\) i.i.d. across responses. In reality, two responses that are nearly identical produce far noisier comparisons than two that differ obviously. That heteroscedasticity turns out to matter a great deal for DPO specifically, and it is exactly the regime where likelihood displacement bites.

The reward model, and its four pathologies

The architecture is almost anticlimactic. Take the supervised fine-tuned model. Delete the unembedding matrix that maps the final hidden state to vocabulary logits. Put a single linear map \(w \in \mathbb{R}^d\) in its place. Run the prompt and completion through, take the hidden state at the last position, and let

That is the entire modification: a language model with a scalar head instead of a vocabulary head, trained with the Bradley–Terry loss instead of cross-entropy. The reward model does not need to be the same size as the policy — InstructGPT used a 6B reward model to train a 175B policy — because predicting a single number about a completed response is a much narrower task than generating one.

The data format has a subtlety that is worth copying. Rather than collecting isolated pairs, you sample \(K\) completions per prompt (InstructGPT used \(K\) between 4 and 9), have a labeler rank all of them, and expand to all \(\binom{K}{2}\) comparisons. Crucially, all pairs from one prompt go into a single batch element. If you shuffle them across batches instead, each completion gets forward-passed \(K-1\) times rather than once, and — more importantly — the model sees the same completion repeatedly in different pairs and overfits after roughly one epoch.

Pathology 1 · The distribution it was fitted on is not the distribution it is used on

This is the structural one, and everything else is downstream of it. The reward model is a supervised model. It has evidence in the region of completion-space covered by its training data — completions from \(\pi_{\rm SFT}\) and a handful of other models. Outside that region it is extrapolating, and nothing in its training told it how to extrapolate honestly.

Meanwhile the policy is being explicitly optimised to increase \(r_\phi\). It will therefore drift toward whichever corner of completion-space the reward model happens to be most optimistic about — which, since the optimism out there is unconstrained by data, is very likely a corner where the reward model is simply wrong. The optimiser is not a neutral consumer of the reward model. It is an adversary conducting a directed search for its errors.

Pathology 2 · Goodhart's law, measured

Gao, Schulman and Hilton put a number on this by building a synthetic setup: a fixed large “gold” reward model stands in for humans and generates labels, a smaller “proxy” reward model is trained on those labels, and the policy is optimised against the proxy while the gold score is monitored. Because the gold model is a model, they could collect unlimited labels and actually fit scaling laws.

They found that gold reward as a function of optimisation distance \(d = \sqrt{D_{\rm KL}(\pi \Vert \pi_{\rm init})}\) follows two different functional forms depending on how you optimise:

Both rise, peak, and fall. The coefficients vary smoothly with the number of reward-model parameters — bigger proxy reward models are more robust, peaking later and higher — while the policy size mattered much less. Best-of-\(n\) sampling and RL apply optimisation pressure at different rates as a function of KL, with RL slower at both optimising and over-optimising, which is itself a caution against treating KL as a currency that lets you compare methods.

Pathology 3 · Length

Reward models systematically prefer longer completions, and length is trivially easy for a policy to increase. Singhal and colleagues examined this directly and found that the length correlation in RLHF is not a minor confound but a substantial fraction of the measured improvement: in several settings, interventions that controlled for length removed most of the apparent gain, and simply training the policy to produce longer outputs reproduced a good deal of the reward increase.

Length is worth dwelling on because it is the cleanest available example of the general failure mode. It is a feature that is (a) genuinely correlated with quality in the training data, (b) far cheaper for the policy to increase than the thing it correlates with, and (c) invisible in the reward number itself. Every reward model has some set of such features. Length is just the one we have learned to name.

Pathology 4 · Calibration and sycophancy

Because reward is identified only up to scale, and because the loss saturates, nothing forces a fitted reward model's margins to correspond to actual preference probabilities. A reward model reporting a margin of 6 on a pair where real annotators would split 60/40 is not more accurate — it is overconfident, and that overconfidence is precisely what a policy will chase.

Sycophancy is the same failure with a human face. Sharma and colleagues showed that human preference data itself rewards agreement: when a response matches the user's stated view, both human annotators and preference models sometimes prefer it over the more accurate response. The reward model is not malfunctioning when it produces sycophantic outputs; it is faithfully reproducing a bias in the labels. You cannot fix that with a better optimiser.

The KL-regularised objective

With a reward model in hand, the RLHF objective is one line:

Maximise reward, but stay near the reference policy. This equation is the single most important object in the article: PPO-based RLHF optimises it by sampling, and DPO is nothing more than the observation that it has a closed-form solution which can be substituted backwards into the Bradley–Terry likelihood.

Solving it exactly

For a fixed prompt \(x\), we are choosing a distribution \(p\) over completions to maximise \(\sum_y p(y) r(y) - \beta \sum_y p(y)\log\frac{p(y)}{q(y)}\), where \(q = \pi_{\rm ref}\). Here are two routes to the answer; the second is shorter and tells you more.

What that equation means, three ways

As reweighting. The optimal policy is the reference policy with every completion's probability multiplied by \(e^{r/\beta}\) and renormalised. It is an exponential tilt, and this is the source of the single most clarifying fact about KL-regularised RLHF: the optimal policy has the same support as the reference policy. If \(\pi_{\rm ref}(y\mid x) = 0\), then \(\pi^\star(y\mid x) = 0\) for every finite \(\beta\) and every reward function whatsoever. RLHF cannot conjure a behaviour the base model would never emit. It can only make rare good behaviour common. In practice nothing has probability exactly zero, but “exponentially small under \(\pi_{\rm ref}\)” and “unreachable” are close enough to the same thing.

As thermodynamics. Read \(-r\) as a potential energy and \(\beta\) as temperature. Then \(\pi^\star\) is the Boltzmann distribution of a gas whose initial density was \(\pi_{\rm ref}\), sitting in the landscape carved by the reward. At high temperature the gas barely notices the landscape. At low temperature it all pools in the deepest well, however narrow. \(\beta \log Z\) is, literally, the free energy.

As Bayes. Rearranged, \(\pi^\star \propto \pi_{\rm ref}\cdot e^{r/\beta}\) is prior times likelihood. Korbak and colleagues made this precise: RL with a KL penalty is better understood as Bayesian inference, with \(\pi_{\rm ref}\) the prior and \(\exp(r/\beta)\) the evidence. This is a genuinely useful reframing, because it explains why the reference policy is not an inconvenience to be minimised away. It is the prior. Remove it and you are doing maximum likelihood on a noisy, misspecified evidence term, which is exactly as bad as it sounds.

Why we cannot just use the formula

We have a closed-form optimal policy. Why is there any more work to do? Because \(Z(x)\) is a sum over the space of all completions. For a 100k-token vocabulary and 500-token responses that is \(10^{2500}\) terms — not large, meaningless. We can neither evaluate \(\pi^\star\) nor sample from it directly.

So classical RLHF does the sampling thing instead: draw completions from the current policy, score them with \(r_\phi\), subtract a per-token KL penalty \(-\beta\log\frac{\pi_\theta(y_t\mid\cdot)}{\pi_{\rm ref}(y_t\mid\cdot)}\), estimate advantages, and take a clipped policy-gradient step. The machinery is PPO and it is expensive: four models resident (policy, frozen reference, reward model, value head), a generation loop in the middle of the training loop, and the usual policy-gradient sensitivity to every hyperparameter you have.

But look again at the last stage of that figure. \(Z(x)\) is intractable, yes — and it also depends on nothing but the prompt. Bradley–Terry only ever looks at differences of rewards within a prompt. An intractable quantity that appears identically on both sides of a subtraction does not need to be computed.

The DPO identity, derived slowly

Rafailov and colleagues' contribution is not a new theory of preferences. It is the observation that the two stages we just built — fit \(r\), then solve \(\max_\pi \mathbb{E}[r] - \beta\mathrm{KL}\) — compose into something with a closed form, and that the composition can be inverted.

Here is the argument in six steps. Nothing in it is deep; the whole thing is a change of variables. What makes it powerful is which change of variables.

Reading the result correctly

The temptation is to describe DPO as “RLHF without the RL.” That is catchy and slightly misleading. A better description: DPO is reward modelling in which the reward model happens to be parameterised as a policy. The quantity in the parentheses,

is a real reward model. You can evaluate it, rank completions with it, and it will do a respectable job. The reason no RL is needed is that this particular reward model is already the optimal policy for itself, by construction — so fitting the reward model and solving the RL problem are the same operation.

Rafailov and colleagues later sharpened this in From \(r\) to \(Q^\star\), showing that under a token-level maximum-entropy formulation the same implicit reward can be read as a \(Q\)-function, which gives DPO a per-token credit-assignment interpretation it does not obviously have in the sequence-level derivation above.

Practical notes

• \(\beta\). Typically 0.01–0.5 for chat models, and it does not transfer across datasets, for the reason established in §3: reward scale is set by annotator agreement, so \(\beta\) is measured in units that change with your labelling pipeline.
• The reference forward pass. \(\pi_{\rm ref}\) is frozen, so its log-probabilities can be precomputed once over the whole dataset and cached, reducing training to a single trainable forward and backward. With LoRA you can go further and use the adapter-disabled base model as the reference, holding only one set of weights.
• Which reference. The SFT checkpoint. If your preference data was generated by a different model, you are already off the derivation's assumptions; the usual mitigation is to run a brief SFT pass on the preferred completions first so that \(\pi_{\rm ref}\) at least covers the data.

What the DPO gradient actually does

Differentiate the loss. Writing \(\hat r_\theta(x,y) = \beta\log\frac{\pi_\theta(y|x)}{\pi_{\rm ref}(y|x)}\) for the implicit reward:

Three factors, each worth reading separately.

The two gradient terms are the obvious ones: increase the log-probability of the preferred completion, decrease the log-probability of the rejected one. This is the negative signal that SFT could never provide.

The scalar in front is the interesting one. \(\sigma(\hat r_l - \hat r_w)\) is large — near 1 — exactly when the implicit reward model currently ranks the pair backwards, and collapses toward 0 once the pair is confidently ordered. It is the same self-annealing weight we derived for the Bradley–Terry loss in §3, which is not a coincidence: DPO is a Bradley–Terry fit, just with a peculiar parameterisation of the score. The loss builds its own curriculum, and this is why DPO is comparatively forgiving about learning rate: the effective step size shrinks automatically as pairs get resolved.

The failure mode hiding in plain sight

Now read the loss again and ask what it constrains. It is a function of \(\hat r_w - \hat r_l\) and of nothing else. There is no term anywhere requiring \(\log\pi_\theta(y_w\mid x)\) to go up. The loss is equally satisfied if the winner's probability falls, provided the loser's falls faster.

This is not a hypothetical. It happens routinely, and Razin and colleagues named it likelihood displacement: over DPO training, the log-probability of the preferred response frequently decreases, sometimes dramatically, while the loss goes down monotonically. Probability mass has to go somewhere, and where it goes is a third region that appears nowhere in the objective.

Two more properties of this gradient are worth flagging before we move on, because they explain most of the variants in the next section.

The offline preference family

There is a very large number of three-and-four-letter acronyms in this area and it is not useful to walk through them one at a time. It is much more useful to notice that almost all of them are choices along five axes, and then read the acronyms off the axes.

1. What is the implicit reward? \(\beta\log\frac{\pi_\theta}{\pi_{\rm ref}}\) (DPO, IPO, KTO), the length-normalised average log-probability \(\frac{1}{|y|}\log\pi_\theta\) (SimPO, RRHF), or the log-odds \(\log\frac{p}{1-p}\) (ORPO).
2. What shape is the loss on the margin? Logistic, hinge, squared, or a prospect-theoretic value function.
3. Is there a reference policy at all?
4. Is the data pairwise or pointwise?
5. Is the data on-policy? The axis that turns out to matter most, and the one the acronyms tell you least about.

The comparator below plots the loss and — more revealingly — the gradient of each family member as a function of the preference margin. The differences between these methods are almost entirely visible in the gradient plot, and almost invisible in the loss plot, which is a small lesson in itself.

The map

The column that predicts benchmark differences best is the third one, not the last two.

A handful of further variants exist to patch specific failure modes rather than to reconceive the objective, and they are worth knowing by function rather than by name: DPO-Positive adds a penalty when the preferred response's log-probability falls below the reference's, directly targeting likelihood displacement; RPO and similar add an explicit negative-log-likelihood term on the preferred completion for the same reason; R-DPO subtracts an explicit length penalty; Cal-DPO and DPO-Shift reshape the implicit reward's calibration; and CPO drops the reference model while keeping the logistic form.

Online versus offline

Strip away the loss shapes and one axis is left standing: where do the completions come from?

The DPO loss is a sum over the pairs in your dataset. It constrains \(\pi_\theta\) at exactly the completions that appear in those pairs, and it is silent everywhere else. In a space of \(10^{2500}\) sequences, “everywhere else” is essentially all of it. So the minimiser of the empirical DPO objective is enormously underdetermined off-support, and the particular solution optimisation lands on is decided by the parameterisation and the initialisation rather than by the data.

PPO does not have this problem, because it evaluates the reward on completions sampled from the current policy. Whatever the policy is doing right now, the reward model gets asked about it. That is a materially different kind of supervision, even when the underlying preference signal is identical.

The empirical literature on this is real but noisier than the theoretical argument suggests:

• Xu and colleagues argued directly that DPO is limited by exactly this distribution shift, and reported PPO outperforming DPO across their benchmark suite when the reward model is good — including on code competition problems, where the gap was large.
• Ivison and colleagues unpacked the comparison more carefully and attributed much of PPO's advantage to reward-model quality and to the extra compute the online loop consumes, finding a smaller gap under matched conditions.
• Tajwar and colleagues located the mechanism more precisely: what helps is on-policy sampling combined with a negative gradient — that is, being able to push down on the model's own currently-likely-but-bad outputs. Neither ingredient alone reproduces the effect.
• Guo and colleagues closed the loop with online AI feedback, generating fresh pairs from the current policy at every step and labelling them with an LLM judge, which recovers much of the on-policy benefit without a reward model.

And then there is what happens when someone runs the ablation carefully on a full recipe. The Tülu 3 team compared preference-tuning algorithms on a matched setup, with tuned hyperparameters, on the same data:

Tülu 3, table 18. Same data, tuned hyperparameters. The spread across algorithms is about two points; the spread across hyperparameters within one algorithm is nearly eleven.

Read that table carefully, because it is the most useful thing in this section. The best algorithm beat the worst by a couple of points. The best configuration of SimPO beat the worst configuration of length-normalised DPO by more than that. And the interventions the same team reported as mattering most were not on this list at all: scaling the number of unique prompts, adding on-policy completions to the mixture, and using new prompts rather than reusing the SFT prompts.

The cost accounting

The other reason DPO took over the open ecosystem has nothing to do with quality. It is that PPO needs the policy, a frozen reference, a reward model and a value head resident simultaneously, plus a generation loop inside the training loop. DPO needs the policy and a frozen reference — and even the reference can be precomputed and cached, or realised as the adapter-disabled base model under LoRA. ORPO and SimPO need one model. That is the difference between a run that fits on the hardware a research group actually has and one that does not.

The practical consensus that has emerged is a hybrid, and it looks like this: supervised fine-tune; run offline preference optimisation on a large, diverse, partly on-policy preference mixture; then, if you have the budget, iterate — regenerate completions from the current policy, relabel them with a reward model or a judge, and run another round. Each round is offline; the sequence of rounds is online. This is what iterative DPO recipes do, and it captures most of the on-policy benefit at a fraction of PPO's engineering cost.

From human feedback to AI feedback

Everything above still bottoms out in a human making a comparison, and that human is expensive, slow, inconsistent, and not an expert in most of the domains the model will be asked about. The next move in the field's history is to replace them.

Constitutional AI

Bai and colleagues proposed a two-phase pipeline in which human supervision is compressed into a short natural-language document — a set of principles — and the model does the rest.

Phase one, supervised. Prompt a helpful-only model with a red-teaming prompt; it produces a harmful response. Then ask the same model to critique that response against a principle sampled from the constitution, and then to revise it in light of the critique. Repeat the critique-revise loop a few times. Fine-tune on the final revisions. The output is SL-CAI.

Phase two, reinforcement. Sample response pairs from SL-CAI. Present each pair to a feedback model as a multiple-choice question — which response better satisfies this sampled principle? — and take the model's log-probabilities over the answer options as a soft preference label. Train a preference model on those labels and run RL against it. The output is RL-CAI.

The headline result was an improvement in the helpfulness/harmlessness trade-off, and a qualitative change worth noting: the resulting models were less evasive. Rather than refusing with a stonewall, they tended to explain their objection, which is what you would expect from a training signal built out of critiques.

RLAIF, measured against RLHF

Lee and colleagues ran the direct comparison. On summarisation, human evaluators preferred RLAIF over the SFT baseline 71% of the time and RLHF over SFT 73% of the time — statistically indistinguishable. On helpful dialogue, 63% versus 64%. Head to head, RLAIF against RLHF was roughly a coin flip. On harmless dialogue, RLAIF actually scored higher: an 88% harmless rate against RLHF's 76% and SFT's 64%. They also introduced direct-RLAIF, which skips the reward model entirely and queries an off-the-shelf LLM for a score inside the RL loop.

One detail from that paper deserves more attention than it usually gets: the reward model trained on human labels had higher pairwise accuracy than the one trained on AI labels — 79.3% against 74.2% on summarisation — and yet the downstream policies came out equivalent. Reward-model accuracy on held-out comparisons is, apparently, a weak predictor of the quality of the policy you get by optimising against it. Which is consistent with everything §4 said about where the reward model actually gets queried.

The obvious caution: an LLM judge is not a neutral oracle. Judges exhibit position bias (preferring whichever response is shown first), verbosity bias (the same length problem as reward models, now in the evaluator), and self-preference (rating their own family's outputs higher). Because the same model family is often used to generate, to label and to evaluate, these biases can be invisible in every metric you are looking at.

The genuine advantage of the constitutional framing is not that AI labels are more accurate. It is that the value judgements become an auditable artefact. A constitution is a document a person can read, argue with, and revise. Ten million preference labels are not.

Where this is going

Step back and look at the trajectory. The reward signal started as a human's direct judgement. It became a learned model of that judgement. Then the labeler itself became a model, guided by a written document. Each step made the signal cheaper and more scalable. None of them made it more correct. Everything in this article has been an exercise in propagating a fixed, finite, noisy quantity of human judgement as far as it will go.

The next article changes the nature of the signal. For a large and growing class of tasks — mathematics with a checkable answer, code with a test suite, formal proofs, instruction-following with a programmatic verifier — the reward is not a model of anyone's preference. It is a function that returns 1 if the answer is right. No Bradley–Terry, no annotator noise, no reward model to over-optimise. Different failure modes appear immediately, and they are the subject of the next piece.

What we actually know

This field moves quickly and publishes confidently, and it is worth separating the parts that have held up from the parts that are still contested. Here is my honest ledger.

Things that appear solid

• Preference tuning after SFT reliably improves human-judged quality. This has replicated across labs, model scales, base models and task families for seven years. Whatever the mechanism, the effect is real.
• Comparisons are better than ratings as an elicitation format. Annotators are more consistent when ranking than when scoring, and the Bradley–Terry machinery converts rankings into the scalar the optimiser needs.
• Reward models degrade under optimisation pressure, predictably. Gao and colleagues' functional forms in \(\sqrt{\rm KL}\) have held up, and the practical consequence — that there is an interior optimum and you should stop before the proxy peaks — is now standard practice.
• The DPO identity is a theorem. Given the KL-regularised objective, given Bradley–Terry, and given the exact optimum, the reward really is recoverable from the policy. That derivation is not contested. What is contested is what happens when the givens fail.
• On-policy data helps, and the effect is not an artefact of loss shape. Tang and colleagues ruled out data coverage, data quality, the contrastive loss and policy scale one at a time, and the gap persisted. Along the way they found something odd worth remembering: offline-trained policies are better pairwise classifiers but worse generators, and online-trained policies are the reverse.
• Length is a confound almost everywhere. In the preference data, in the reward model, in the DPO objective by construction, and in the LLM judges used to evaluate all of it.

Things that are genuinely open

• Which offline loss is best. There is no reliable ordering. Under matched data and honestly tuned hyperparameters, the family lands within a couple of points of each other and the ranking flips between papers. Every headline claim of a large margin that I have looked at closely dissolved into a hyperparameter or a data difference.
• Whether Bradley–Terry is the right model at all. It assumes a total order over responses, homogeneous annotators, and i.i.d. noise. All three are false. The Nash-learning line of work drops the scalar-reward assumption entirely and looks for an equilibrium of the preference relation instead; whether that matters in practice is not yet settled.
• What KL divergence from the reference is actually measuring. We use it as a proxy for “still a competent language model”, and it is a poor one. Two policies at identical KL can differ enormously in how degenerate they are, and \(\beta\) is not a KL budget in any reliable sense.
• Whether preference tuning teaches capability or elicits it. The superficial-alignment hypothesis — that essentially all capability is learned in pretraining and alignment only selects a format — is a hypothesis with mixed evidence, not a finding. Small curated SFT sets do get surprisingly far, which supports it; but RLHF also produces measurable gains on tasks with correct answers, which does not. I would treat anyone who states either side flatly as overclaiming.
• How to evaluate any of this. The dominant metrics are win rates judged by strong LLMs, which inherit position bias, verbosity bias and self-preference. We are optimising against a proxy and evaluating against a correlated proxy, which is exactly the configuration in which Goodhart damage is hardest to see.

What to carry away

If you retain four things from this article, make them these.

One. The entire edifice exists because “good” has no derivative. Every technique here is a way of manufacturing one: a comparison becomes a probability via Bradley–Terry, a probability becomes a scalar field via a learned model, a scalar field becomes a policy via KL-regularised maximisation. Each step is a place where information is lost and assumptions are inserted.

Two. The KL term is not a regularisation afterthought. It is what makes the objective have a closed-form solution, what defines the trust region the reward model is valid inside, and what makes the DPO derivation possible at all. Remove it and both the mathematics and the engineering fall apart.

Three. DPO's elegance is real and its scope is narrower than it looks. The identity holds at the exact optimum, under Bradley–Terry, on the data you have. Real training is none of those things, and the gap between the theorem and the practice is exactly where likelihood displacement, length blow-up and off-support underdetermination live.

Four. The data pipeline dominates the loss function. Prompt diversity, label quality, on-policy completions and length control move the numbers reproducibly. The choice of acronym moves them by about a point. Spend accordingly.

The uncomfortable thread running through all of it is that we never escaped the original problem. We are still, at the end, optimising against a model of what a tired annotator preferred on a Tuesday. The next article is about what changes when the reward stops being a model of anyone's preference and starts being a fact.

References

Grouped by the argument they support rather than alphabetically. Every link goes to the primary source.