How Reinforcement Learning Teaches Models to Reason

Two kinds of reward

Every reinforcement learning system is defined by the thing it is allowed to measure. Change the measurement and you have not tuned the method, you have changed the problem. The whole story of reasoning RL is the story of one substitution in one symbol.

In the RLHF pipeline covered in the previous post, the reward is a learned function. You collect pairs of model responses, ask a human which one they prefer, fit a Bradley–Terry model to those preferences, and end up with a network rψ(x, y) that outputs a scalar. That scalar is a guess about what a human annotator would have said. It is a model of a taste.

In reasoning RL the reward is a function you write down:

That is the whole idea. The name attached to it is RLVR, reinforcement learning with verifiable rewards, which was coined in the Tülu 3 report, where the authors describe RLVR as reusing the existing RLHF objective but replacing the reward model with a verification function, assigning a fixed positive reward when the answer is verifiably correct and zero otherwise. Written out, the objective is the RLHF objective with one symbol swapped:

It is worth being precise about what that substitution actually buys, because the popular framing — "now the reward is objective!" — misses the operationally important parts.

The last row is the one that keeps RLHF alive. A verifier can tell you the answer is 42. It cannot tell you whether the explanation was condescending. RLVR did not replace RLHF; it opened a second, cheaper channel next to it.

The fourth row is the one that made the field move. Reward hacking under a learned reward model is not a hypothetical failure — it is the default outcome of optimizing hard enough against any learned proxy. A verifier is not immune, but it is qualitatively harder to fool, because the failure mode requires exploiting a real program rather than an interpolation gap in a neural network.

One caveat that the rest of this article keeps returning to. The indicator 1[answer correct] is a statement about the last few tokens of a four-thousand-token trajectory. It says nothing whatsoever about the reasoning that produced them. A model that reaches the right answer through a sign error that cancels another sign error receives exactly the reward that a flawless derivation receives. We are optimizing a proxy for reasoning, not reasoning. That proxy is cheap, scalable and hard to fool — and it is still a proxy.

The verifier era

Reasoning RL did not appear out of nowhere in 2025. It rests on an observation made four years earlier, in a paper whose main contribution is usually remembered as a dataset.

Cobbe et al. (2021) introduced GSM8K, and with it a procedure: instead of fine-tuning a model to produce one good solution, train a separate verifier to judge the correctness of model completions, sample many candidate solutions at test time, and return the one the verifier ranks highest. The headline number is worth sitting with. On the full dataset, verification with a 6B model slightly outperformed a fine-tuned 175B model, a boost roughly equivalent to a 30× increase in model size, and verification scaled better with additional data than the fine-tuning baseline.

Read that structurally rather than as a benchmark result. It says that generating a solution and recognizing a good solution are separable computations with different cost curves. And it says the recognizer is the cheaper one.

Once you accept the asymmetry, two different products fall out of it depending on when you spend the verification budget.

• Spend it at inference. Sample N solutions, score them, return the best. This is best-of-N, and it is the ancestor of everything in test-time scaling. The model's weights never change; you have simply bought accuracy with FLOPs.
• Spend it during training. Sample N solutions, score them, and use the scores as a learning signal so the model becomes more likely to produce the good ones on its own. This is RLVR. You have bought a permanent shift in the policy.

These are the same operation applied at different times, which is why the two literatures keep colliding. It is also why the arguments in section nine — does RL create new capability or concentrate existing capability? — are so hard to settle. Both spend the same budget on the same distribution.

The 2021 paper also contained the first published sighting of the failure mode that has haunted the field ever since. Reranking accuracy improves as you increase the candidate pool, up to a point, and then turns over: past a few hundred ranked completions, the search starts finding solutions that fool the verifier rather than solutions that are correct. Optimization pressure against an imperfect scorer eventually optimizes the scorer's errors. Hold that thought until section three.

Outcome and process

If a verifier is useful, a more informative verifier should be more useful. That reasoning produced the outcome-versus-process debate, which is the most theoretically satisfying part of this literature and also the part where practice has most stubbornly refused to follow theory.

The credit assignment argument

Consider a solution with five reasoning steps and a final answer. Outcome supervision gives you one bit at the end. Process supervision gives you one bit per step.

Two distinct advantages follow, and it helps to keep them separate because they fail in different ways.

Variance. The policy gradient estimator for a single terminal reward must attribute one scalar across every token in the trajectory. Every token in a correct 4,000-token solution gets pushed up; every token in an incorrect one gets pushed down, including the many tokens that were perfectly fine. The signal is real but the noise scales with length. Step-level rewards localize the credit and shrink the variance.

Alignment of the objective with the thing you want. Outcome supervision positively reinforces false-positive trajectories — the ones that arrive at the right answer through broken reasoning. Under enough optimization pressure, "lucky" becomes a strategy. Process supervision, in principle, closes that door.

Uesato et al. (2022) ran the first careful head-to-head and found the two methods reached similar final-answer error rates, with process supervision achieving them from less data and producing cleaner traces. Lightman et al. (2023) scaled that comparison up and reported a much sharper result: process supervision significantly outperformed outcome supervision for training models to solve problems from the MATH dataset, with the process-supervised model solving 78% of problems from a representative subset of the MATH test set. They also released PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train the best reward model.

The paper's stated motivation is worth quoting in spirit rather than in full: process supervision gives more precise feedback because it names the location of the error, it is easier for humans to interpret, and it rewards the model for following a chain of reasoning a human would endorse, rather than merely arriving somewhere a human would endorse. Those are alignment arguments as much as capability arguments.

Why the field went back to outcome rewards anyway

Given that evidence, the natural expectation in early 2025 was that frontier reasoning systems would be trained against process reward models. That is not what happened, and the reasons are instructive.

The first problem is the labels. Human step-level annotation does not scale; PRM800K cost a great deal to produce and covers one domain. The obvious substitute is Monte Carlo estimation: roll out many completions from each intermediate step and score the step by how often those completions reach the right answer. The Qwen team's "Lessons of Developing Process Reward Models" examined this carefully and found that Monte Carlo estimation-based data synthesis typically yields inferior performance and generalization compared with LLM-as-a-judge and human annotation, because the completion model can generate correct answers from incorrect steps and incorrect answers from correct steps, which makes the step verification itself inaccurate.

That is a subtle and important point. The MC estimator asks "can the model recover from here?" and treats the answer as "was this step correct?" Those are different questions. A strong completion model rescues bad steps; a weak one botches good ones. Your step labels end up encoding the completion model's competence rather than the step's validity.

The second problem is evaluation. The same paper identifies biases in how PRMs are usually judged: unreliable policy models produce responses with correct answers but flawed processes, which misaligns best-of-N evaluation with the PRM's actual objective; PRMs tolerate those responses, which inflates best-of-N scores; and existing PRMs concentrate a large share of their minimum scores on the final answer steps, revealing a drift from process assessment toward outcome assessment. In other words, PRMs optimized for best-of-N quietly turn back into outcome reward models.

The third problem is the one Cobbe's turnover curve warned about. A PRM is a learned scorer, and once you put it inside a training loop rather than a reranking loop, the policy has millions of gradient steps to find its blind spots. The DeepSeek-R1 report is explicit that they abandoned process reward models for large-scale RL for exactly this combination of reasons: the difficulty of defining fine-grained steps, the unreliability of automatic step annotation, and the reward hacking that follows from putting a neural reward model in the loop, together with the cost of continually retraining it.

None of this means process supervision was a wrong turn. It means the field found a cheaper thing that survives optimization pressure, and took it. Process supervision remains the more principled objective and remains an active research direction, particularly for domains where outcome verification is unavailable or where you care about the derivation for its own sake. But the systems that produced the 2025 capability jump were trained against an indicator function on the last line.

What PPO costs you

Before the algorithm that everyone now uses, it helps to be concrete about the algorithm it displaced, and about what specifically was wrong with it. Not "PPO is complicated" — that is a vibe, not an argument. Something sharper.

Four networks in memory

A textbook PPO-based RLHF loop holds four models:

Put numbers on that. Take a 7B policy with a 7B critic, mixed-precision training, Adam. Following the accounting from Inside a Training Step, a trained model costs roughly 16 bytes per parameter once you count bf16 weights, bf16 gradients, the fp32 master copy, and Adam's two moments. A frozen model in bf16 costs 2.

That is a systems problem and it is real, but it is not the deepest problem. The deepest problem is that the critic is being asked to do something close to impossible.

The critic has an unlearnable job

V_φ(s_t) is supposed to estimate the expected return from state s_t. In a long chain-of-thought setting, s_t is "the prompt plus the first 1,200 tokens of a partial derivation." The critic must answer: given that the model has written this much, how likely is it to finish correctly?

Three things make that hard.

• The target is almost binary and almost entirely determined at the end. A derivation can look immaculate for 1,200 tokens and then divide by zero. The value function has to encode the entire remaining difficulty of the problem plus the model's own future reliability, from a prefix.
• The training signal is a single terminal reward. Every intermediate value target is bootstrapped from that one scalar. Value initialization bias and the choice of GAE parameters end up dominating training dynamics, which is precisely what later work on value-calibrated PPO variants had to fix.
• The reward is terminal anyway. There are no intermediate rewards to propagate. The critic is not providing new information about the environment; it is providing variance reduction. That is a real service, but it is worth asking what a much cheaper variance reducer would cost.

What you can delete

Ahmadian et al. (2024) made the argument directly. PPO's machinery exists to solve problems that arise in classic deep RL: unstable policies initialized from scratch, per-step rewards, many optimizer epochs over the same data, and the risk of catastrophic policy collapse. LLM post-training has none of those properties. You start from a strong pretrained prior, the reward arrives once at the end, and you typically take one inner epoch per batch of rollouts. Their conclusion was blunt: many components of PPO are unnecessary in this setting, and simpler REINFORCE-style variants match or beat it. Modelling partial completions, in particular, turned out to be unnecessary — treating the whole generation as one action performed as well or better.

That reframing matters more than the benchmark table. If the whole generation is a single action and the reward arrives once, then this is not a sequential decision problem at all. It is a contextual bandit: you see a context x, you take one action o, you receive one scalar r. Everything about PPO that exists to handle sequential credit assignment is dead weight.

Their concrete proposal, RLOO, is the last stepping stone before GRPO. Sample k completions per prompt. For completion i, use the mean of the other k−1 rewards as its baseline:

GRPO from first principles

Group Relative Policy Optimization was introduced in DeepSeekMath in February 2024, eleven months before it became famous. The paper describes it as a variant of PPO that forgoes the critic and estimates the baseline from group scores, which significantly reduces training resources. That is accurate and it undersells the elegance. Let us derive it rather than quote it.

Step one: the gradient of a bandit objective

We want to maximize expected reward under our own policy:

Differentiate. The only θ-dependence is in the sampling distribution, so we use the log-derivative trick, ∇π = π ∇logπ:

This estimator is unbiased and its variance is terrible. If every reward in your batch is +1 — which happens constantly when a model is good at a problem — you push up the log-probability of everything, learn nothing about which sample was better, and take a large, uninformative step.

Step two: baselines are free

Subtract any quantity b that does not depend on o. The gradient is unchanged:

This four-line identity is the entire justification for critics, for group means, for leave-one-out baselines, and for GRPO. It is the free lunch of policy gradient methods: you may subtract whatever you like as long as it does not peek at the sample it is scoring.

So the design question becomes: what is the best b to subtract? The variance-minimizing choice is a gradient-norm-weighted mean of the rewards, which nobody computes. The practical choice is the expected reward, E[r | x]. In the sequential formulation that quantity is the value function V(x) and you learn it with a network. In the bandit formulation, look at what it actually is:

Step three: the group is the baseline

That is GRPO. For each prompt x, sample a group of G outputs from the current policy, score them, and use the group's own statistics to center and scale:

Slot that into a clipped surrogate objective and you have the DeepSeekMath formulation:

The KL term uses the unbiased, always-positive k3 estimator rather than the naive log-ratio, and it appears as a separate term in the loss rather than being folded into the reward as PPO-based RLHF did:

What the algebra tells you about binary rewards

Because the reward is an indicator, the group statistics have closed forms. Let p be the fraction of the group that got the answer right. Then μ = p and σ = √(p(1−p)), so:

Three consequences fall straight out, and each one has a paper attached to it.

Rare outcomes dominate. At p = 7/8 — an easy problem where the model fails once out of eight — each correct sample gets A = √(1/7) ≈ 0.38 while the single failure gets A = −√7 ≈ −2.65. The one mistake carries seven times the weight of each success. Symmetrically, on a hard problem where one sample out of eight succeeds, that lone success gets an enormous positive advantage. The z-score normalization automatically turns up the gain on whichever outcome is surprising, which is a reasonable thing to want and also a source of instability when a group is nearly unanimous.

Unanimous groups are wasted compute. At p = 0 or p = 1, σ = 0 and every advantage is 0/0. Implementations add an ε to the denominator, which makes the advantages approximately zero, which means the prompt contributes no gradient at all. You paid for eight full rollouts — thousands of tokens of generation each — and bought nothing. This is why DAPO's dynamic sampling, which resamples until a batch contains groups with non-degenerate accuracy, is a throughput intervention and not only an accuracy one. It is also why curriculum and difficulty filtering matter far more in RLVR than the algorithm papers suggest.

Dividing by σ introduces a difficulty bias. Two prompts that produce the same information — say, one where 4/8 succeed and one where 1/8 succeeds — contribute gradients of very different magnitude, purely because of the shape of the √(p(1−p)) denominator. Whether you want that is a modelling choice, and GRPO makes it silently. We will come back to it.

The one honest wrinkle

The group mean includes r_i itself, so strictly speaking b is not independent of the sample it scores, and the estimator picks up an O(1/G) bias. RLOO's leave-one-out baseline removes it exactly. In practice with G = 8 to 64 nobody has found this to matter, and the difference between GRPO and RLOO in the limit of large G is nil. It is worth knowing about mostly because it tells you which of the two derivations is the clean one.

R1-Zero and R1

In January 2025 DeepSeek released two models and one report. The report is short, unusually candid about what did not work, and is routinely summarized in a way that erases its most important distinction. So: the distinction first.

Two different pipelines

Collapsing both into "pure RL" is the single most common error in write-ups of this work. R1-Zero is the scientific result: reasoning behaviour can be incentivized by RL alone, starting from a base model, without any human-written reasoning trajectories. R1 is the engineering result: here is how you make that usable, and it involves four stages, two of which are supervised.

The reward for R1-Zero had two components and no neural network in either:

Three things happened during that training run, all of which the report documents.

Response length grew on its own. Nobody asked for longer chains of thought. No length term appeared in the reward. Yet average response length climbed steadily across training as accuracy climbed. The natural reading — and the report's reading — is that the model discovered it could spend more test-time compute per problem, and that spending it paid.

Reflective behaviours appeared. Self-verification, backtracking, and re-derivation showed up in the traces, including the now-famous table in which an intermediate checkpoint writes something to the effect of stopping mid-derivation to reconsider its own step. The authors call this an aha moment and describe it as the model developing problem-solving strategies rather than being taught them.

The distilled models beat the RL'd small models. Alongside R1 the team released six dense models distilled from R1's outputs, and reported that distilling R1 into a small model produced better reasoning than running the same large-scale RL on that small model directly. This is a quiet but very large clue about where the capability is coming from, and we will pick it up in section nine.

What actually changed

It is worth separating what R1 demonstrated from what R1 caused.

• It made the recipe credible and public. o1 had already shown that RL on reasoning worked; OpenAI's blog post said nothing about how. R1 published a method, a reward specification, and weights. Nearly every paper in section eight exists because those weights and that recipe were downloadable.
• It moved test-time compute from a prompting decision to an optimization target. Chain-of-thought had been something you asked for. After R1, the length of the chain became something the optimizer chose.
• It normalized GRPO. Not because GRPO is uniquely good — section eight has several counterexamples — but because it was the algorithm attached to the result.

And what it did not establish: that RL invented reasoning. That claim is much stronger than the evidence, and within three months of publication several groups had produced results that complicate it considerably.

Aha moment or optimization artifact?

This is where a careful reading diverges from the popular one. Four independent lines of work, all from 2025, each chip away at a different part of the emergence story. Taken together they do not overturn it. They relocate it.

The base model already did that

Liu et al., "Understanding R1-Zero-Like Training: A Critical Perspective" examined the two ingredients separately — base models and RL — and found that the base model contributes more than the framing suggested. DeepSeek-V3-Base already exhibits aha-moment language before any reinforcement learning is applied. And Qwen2.5 base models answer mathematical questions competently with no prompt template at all, which strongly suggests they were pretrained on question-answer-formatted data. If the base model already reflects, backtracks and answers questions, then "RL elicited these behaviours" and "RL created these behaviours" are very different claims, and the evidence supports only the first.

Part of the length curve is a normalization bug

The same paper identifies an optimization bias in GRPO that artificially inflates response length, especially for incorrect outputs. It is worth deriving because it is a genuinely instructive example of how a normalization constant becomes a behavioural prior.

Look again at the objective. Each response i contributes a term normalized by its own length:

Every token in response i is pushed by the same amount, |A_i|/|o_i|. Now consider two incorrect responses in the same group, so both have the same negative A_i. One is 200 tokens, one is 1,400. The short one has each of its tokens pushed down seven times harder than the long one.

The optimizer does not know it is being asked about correctness. It sees a loss surface on which, among the responses it is being told to avoid, the long ones are cheaper to keep. Repeat that for thousands of steps and the length distribution of incorrect responses drifts right. Liu et al. observe exactly this: GRPO continues to generate longer responses even after reward improvement has flattened, and their fix — removing the length normalization and the standard-deviation normalization, which they call Dr. GRPO — prevents the model from generating progressively longer incorrect responses, improving token efficiency without hurting accuracy.

DAPO arrived at the same family of fix independently, replacing per-response normalization with a token-level loss that normalizes by the total token count across the group so that every token in the batch carries equal weight.

The behaviours do not track the length

SimpleRL-Zoo ran zero RL training across ten base models spanning families and sizes — Llama3-8B, Mistral-7B and 24B, DeepSeek-Math-7B, and the whole Qwen2.5 range from 0.5B to 32B. Their observation is the useful one: increased response length does not always correlate with the emergence of cognitive behaviours such as verification. They also note that the Qwen2.5 series, which almost the entire replication literature was built on, is not representative, because those base models already exhibit strong instruction-following and self-reflection. Notably, they report observing an aha moment for the first time in small models outside the Qwen family — which is only a notable result if you appreciate how much of the field had been running on one model family.

What the base model needs to have

Gandhi et al. asked the sharpest version of the question: why does Qwen-2.5-3B improve dramatically under RL on Countdown while Llama-3.2-3B plateaus under identical training? They isolate four cognitive behaviours — verification, backtracking, subgoal setting, and backward chaining — and show Qwen exhibits them natively while Llama largely does not.

Then they intervene. Priming Llama with a small amount of fine-tuning on traces that display those behaviours unlocks RL gains it otherwise cannot achieve, matching Qwen. And the detail that makes the result: priming with traces containing the right behaviours but the wrong answers works just as well as priming with correct solutions. Length-matched chain-of-thought controls without the behaviours fail. Finally, filtering OpenWebMath to amplify these behaviours and continuing pretraining on it lets Llama match Qwen's self-improvement trajectory outright.

The mechanism this suggests is clean: RL amplifies cognitive behaviours that are already latent in the policy. If they are not there, RL has nothing to amplify, and no amount of verifier signal will conjure them.

The reward may not even need to be right

The most destabilizing result in this cluster is Spurious Rewards. The authors ran RLVR on Qwen2.5-Math-7B with rewards that carry little, no, or negative correlation with correctness, and measured absolute gains on MATH-500:

A random reward recovers roughly three quarters of the gain from ground truth. That number should make anyone uncomfortable who believes the verifier is doing the work.

Two qualifications keep it from being a reductio of the whole field. First, the effect is model-specific: spurious rewards that work on Qwen frequently produce nothing on Llama3 or OLMo2. Second, the authors offer a mechanism. Code reasoning — producing code-like reasoning without executing it — is a distinctive Qwen2.5-Math behaviour, and its frequency rises from around 65% to over 90% after RLVR, even under spurious rewards. Their hypothesis is that in the absence of a useful reward signal, RLVR is surfacing reasoning representations already learned during pretraining. They are explicit that the exact mechanism is still open.

Putting it together

The honest synthesis is narrower than the headline and still substantial:

Large-scale RL against a verifier is a remarkably effective way to make a base model reliably do, on the first attempt, things it could previously only do sometimes. The reflective behaviours it appears to discover were largely present in the pretrained policy. Part of the observed chain-of-thought lengthening was an artifact of the loss normalization. And how much of the improvement survives when you change the model family is an open empirical question that the field spent most of 2025 not asking.

That is a less romantic story than "RL taught the model to think." It is also the story that predicts the field's actual behaviour since: an enormous amount of effort going into pretraining data curation and mid-training, on the theory that RL cannot amplify what is not there.

The open lineage

There are hundreds of R1 replications. Most tell you nothing you cannot get from the report. These six are worth knowing because each one isolates a variable.

Why the field went this way. Once verifiable rewards made the objective simple, the bottleneck moved from inventing richer reward models to keeping exploration alive and estimating gradients correctly. The strongest follow-ups therefore vary one practical axis at a time—context length, critic choice, training domain, staging, sampling, or normalization. Together they suggest that reliable RLVR is an engineering problem around a simple signal, not a parade of unrelated algorithms.

Across these runs, three principles recur. The optimizer label is secondary: PPO-like and group-relative estimators can both work when they receive informative batches. Verifiability enables transfer: even narrow synthetic tasks can teach reusable habits such as checking and backtracking when success is unambiguous. Staging controls interference: math and code rewards can be learned first, then mixed with general alignment signals without necessarily erasing the earlier gains.

Engineering fixes that recur across methods

Elicit or create?

Everything so far points at a single unresolved question. When RLVR improves a model, has it acquired a capability the base model did not have, or has it become reliable at a capability the base model already had?

This is not philosophy. It determines whether RL compute is a substitute for pretraining compute or a multiplier on it, and those two worlds have completely different roadmaps.

The measurement that made it a fight

Yue et al. proposed a clean test. Do not measure accuracy at one sample. Measure pass@k at large k — the probability that at least one of k samples is correct — which is a coverage metric. If RL created new reasoning ability, the RL model should dominate the base model at every k. If it merely concentrated probability mass, the base model should catch up and eventually overtake as k grows, because breadth is exactly what concentration costs you.

Across model families, RL algorithms, and math, code and visual reasoning benchmarks, they found the second pattern. RLVR-trained models beat their base models at small k, base models achieve higher pass@k at large k, and the crossover is robust. Worse for the optimistic reading, the reasoning boundary narrows as RLVR training progresses. Coverage and perplexity analysis indicated the reasoning paths generated by RLVR models were already inside the base models' sampling distribution. Treating the base model as an upper bound, their quantitative analysis found six popular RLVR algorithms performing similarly and all remaining far from fully exploiting the base model's potential.

Two details prevent this from being explained away. Raising the sampling temperature to restore the RL model's lost entropy recovered some of the high-k gap but not all of it, so diversity loss is part of the story and not the whole story. And the effect held at every model size tested.

The sharpening picture

Here is the mental model that makes the result unsurprising. The base model puts probability mass across many solution strategies for a given problem, most of them wrong, a few of them right. Sampling 1,024 times explores that distribution broadly and will occasionally stumble onto a rare correct path. RL with a verifier moves mass from strategies that fail onto strategies that succeed. It is a lens, not a light source: it focuses what is already there, and the light it focuses comes out of the tails.

That is exactly what the distillation result from R1 hinted at. If RL were creating capability, running RL on a small model should be competitive with distilling from a large RL'd model. It was not. Capability transferred better through supervised imitation of a stronger model's outputs than through the small model's own exploration, which is what you would expect if exploration is bounded by the base policy's support.

The counterargument

ProRL takes the opposite position and has evidence for it. Their claim is that the "RL only sharpens" finding is an artifact of how short typical RL runs are and how weakly they control entropy. With prolonged training — KL divergence control, periodic reference policy resetting, and a diverse suite of tasks rather than math alone — they report RL-trained models consistently outperforming base models across pass@k, including on problems where the base model fails entirely regardless of how many samples you draw.

Both results can be true. They are measuring different training regimes. The reconciliation that seems most defensible:

• Short RL runs with weak entropy control sharpen and lose coverage. This describes most of the published replications, and Yue et al. characterize them accurately.
• Long RL runs with explicit entropy and KL management, on diverse tasks, can move the boundary. This is expensive and finicky and describes far fewer runs.
• pass@k at large k is a coverage metric that is sensitive to temperature, to the prompt template, and to how you define "solved." It is the right instrument for this question and it is not a clean one.

I would not tell you the question is settled. I would tell you that anyone who states it confidently in either direction is quoting one of these two papers and not the other.

SFT and RL as operators

Which brings us to the sentence that has done the most damage to clear thinking in this area: SFT memorizes, RL generalizes.

The paper behind it, Chu et al. at ICML 2025, is careful and its results are real. On GeneralPoints, an arithmetic card game, and V-IRL, a navigation environment, RL trained with an outcome-based reward generalized across unseen rule variants and unseen visual variants, while SFT tended to memorize the training rules and struggled out of distribution. Their analysis found RL improving the underlying visual recognition capability, not just the policy over it. The paper also says something the slogan drops: SFT remains essential, because it stabilizes output format and thereby enables the RL that follows.

The trouble is what happened to the finding downstream. It got compressed into a claim about the two methods in general, and subsequent work has been steadily eroding that generalization. Follow-ups point out that the SFT arms in these comparisons were often handicapped in specific, fixable ways: trained without long chain-of-thought supervision, run for short durations, built on data of uneven quality, or run on small early-generation base models where capability was the binding constraint. Analyses in 2026 have argued that reasoning-SFT generalization is conditional on optimization duration, data quality, and base-model capability, rather than simply absent.

One of my favourite results in that line looks at which traces you distill. Comparing student models trained on R1-generated chains against gpt-oss-generated chains, the R1 data yields much lower SFT training loss and notably worse downstream generalization. The proposed explanation is structural: R1 frequently branches into alternative reasoning paths without deep deduction, while the other teacher produces more convergent deductive chains, and students inherit the pattern. Filtering out the most heavily branching trajectories improves the student. That is a finding about reasoning structure, not about SFT as a method.

A better question, and a derivation that answers part of it

Instead of asking which method generalizes, ask: what inductive bias does each impose on the update? There is a clean way to see this, because SFT is a policy gradient method with a particular implicit reward.

Start with the SFT objective and its gradient:

Now change measure onto the policy's own distribution, which is where policy gradients live. Multiply and divide by π_θ:

So supervised fine-tuning is policy gradient ascent with the implicit reward

Sit with that. The implicit reward is an inverse probability weight. The less likely the demonstration is under the current policy, the larger the reward the update assigns to it. Applied per token, a demonstration token the model currently assigns probability 0.001 receives an effective reward a thousand times larger than one it assigns probability 1.0. This is precisely the "problematic implicit reward structure" identified in the ICLR 2026 work on rectifying the SFT objective, whose fix is to multiply the token-level gradient by a detached copy of π_θ, cancelling the 1/π and turning the implicit reward into a constant. Doing so recovers a meaningful share of the generalization gap.

Two properties of SFT follow directly from that expression, and both are inductive biases rather than defects.

• Enormous, high-variance updates concentrated on surprising tokens. The tokens driving the SFT gradient are the ones the model finds least likely, which are disproportionately the idiosyncratic choices of the teacher rather than the reasoning content. This is a plausible mechanism for why students inherit a teacher's stylistic branching patterns.
• No negative signal. The implicit reward is non-negative everywhere. SFT can push things up; it cannot push anything down. Whatever the model does that is wrong but not covered by the dataset is untouched. RL's advantage, by contrast, is signed, and roughly half of a group's samples receive a negative one.

Set the two operators side by side:

Framed this way the empirical results stop conflicting. SFT reaches further and lands harder; RL is gentler and cannot leave the neighbourhood. Chu et al. found RL generalizing better in a setting where the needed behaviour was within the base model's reach and the SFT data was narrow. Gandhi et al. found SFT indispensable in a setting where the needed behaviour was outside the base model's reach — priming Llama with cognitive behaviours it did not have was exactly a case of using SFT to move support so that RL had something to amplify.

What we actually know

A summary that tries to be honest about confidence levels, since this literature is eighteen months old and moving.

Well supported

• Cheap, hard-to-fool feedback is the real innovation. Not the algorithm. Replacing a learned reward model with a program removed the annotation bottleneck and most of the reward-hacking surface in one move, and everything downstream follows from that.
• Critics are optional at this scale. In a terminal-reward bandit, the value function at the prompt is just the mean reward at the prompt, and G samples estimate it for free. GRPO, RLOO, and vanilla PPO with λ = γ = 1 are all reaching for the same Monte Carlo baseline by different routes.
• Most of the 2025 algorithmic progress was normalization repair. Length normalization, standard-deviation normalization, clipping asymmetry, and the level at which the importance ratio is defined. Each of these was a bug with a behavioural signature, and fixing them accounts for a large share of reported improvement.
• The base model is the binding constraint. Priming experiments, spurious-reward results, and cross-family comparisons all converge on this. RL amplifies; it does not conjure.

Genuinely unsettled

• Whether RL expands the reasoning boundary. Short runs sharpen; long, entropy-controlled runs may expand. The measurement instrument is itself contested.
• How much of the reported gains survive a change of model family. A field that ran most of its experiments on one family and then discovered that random rewards work on that family should be humble here.
• Whether process supervision comes back. It is the better objective and the harder one to optimize against safely. Progress on trustworthy step-level signals would change the calculus quickly.
• What "reasoning" is being optimized at all. The reward is an indicator on the final answer. Everything the model does in between is unconstrained. That gap is the subject of the next post in this series.

Where to go next

If you want the policy-gradient foundations underneath section five, they are in RL for LLMs from First Principles. The preference-modelling half of the story, including why DPO exists and where it stops working, is in From RLHF to DPO. What happens when you spend the verification budget at inference rather than during training is Test-Time Scaling. And if you are actually going to run one of these loops, the systems problem — interleaving a generation engine with a training engine across a cluster, and keeping their weights in sync — is a different beast entirely, closer to The Parallelism Playbook and The Serving Playbook than to anything in this post.

The last thing worth saying is that the substitution this article opened with has not finished playing out. We swapped a human judge for a calculator, and got scale. The domains where that swap is available — mathematics, code, formal logic, anything with a checker — are exactly the domains where reasoning models have improved fastest, and that correspondence is not a coincidence. The interesting question for the next few years is not how to optimize better against verifiers. It is what to do about everything a verifier cannot see.

References

Grouped by topic. Every entry links to its primary source.