Where supervision runs out
Everything you have read in this series so far has rested on a quiet assumption: that for every input there exists a correct output, written down somewhere, that the model can be pushed toward. Pretraining has it — the next token in the corpus is the target. Supervised fine-tuning has it — a human wrote the demonstration. The loss is a comparison, and comparison needs two things to compare.
Supervised fine-tuning gives you pairs (x, y*) and minimises the negative log-likelihood of the demonstration:
This is a remarkably strong signal. At every one of the thousands of positions in a training sequence, the loss tells you the exact coordinate in the vocabulary simplex that should have received more mass. The gradient is not an estimate of anything. It is the gradient.
Now change one thing. Suppose nobody wrote down y*. Suppose instead you let the model write its own answer, and afterwards some process — a unit test, a symbolic checker, a human, another model — looks at the finished text and returns a single number.
Three things just broke at once, and it is worth being precise about which three, because the rest of this article is a sequence of repairs to exactly these.
The target is gone. There is no y* to take a log-probability of. Cross-entropy is not merely inconvenient here; it is undefined, because it needs a label.
The signal is scalar and terminal. One number arrives after the last token. A 900-token proof that ends in the right answer gets a 1. So does a 40-token lucky guess. The number says nothing about which of the 900 decisions deserved it.
The data distribution now depends on the parameters. In supervised learning the dataset is fixed; you sweep over it. Here the model produces the very samples it learns from. Change θ and you change what you will see next. The training set is a moving target that you are moving.
Figure 1 puts the two regimes next to each other. Watch what the loss has access to in each.
Reading it. The camera zooms to a single position in stages 2–3 so you can see the whole vocabulary distribution and the single coordinate the SFT gradient pushes on. From stage 4 the same prompt is run in reward mode: the model samples, the sequence completes, and only then does a scalar arrive. Autoplay pauses while you step manually and resumes seven seconds after you stop.
Stage 6 is the crux. Two rollouts from the same prompt, one scoring 1 and one scoring 0, and no information anywhere about which token in either sequence made the difference. This is the credit assignment problem, and it is not a detail — it is the whole subject.
You cannot ask "what should token 47 have been?" because nothing knows. But you can ask a weaker, answerable question: "across many sampled answers, does the presence of this token correlate with higher reward?" That substitution — from a pointwise target to a distributional comparison — is the single conceptual move that makes everything that follows possible. Everything else is variance reduction.
There is also a subtler reason to want this machinery, beyond the cases where a target genuinely does not exist. Even when demonstrations are available, they teach the model to imitate a particular way of being right. They never tell it that one of its own habits is bad. A demonstration says "do this"; it cannot say "not that." Reward can say both, and it can say them about text the model actually produces rather than text a human produced. We will come back to what that does and does not buy you in §13.
A language model is already a policy
Before deriving anything, we need to say what kind of object we are optimising. The claim of this section is deliberately deflationary: nothing needs to be added to a language model to make it a reinforcement learning agent. It already is one. We only have to notice.
A Markov decision process is five things: a state space, an action space, a transition rule, a reward, and a policy that maps states to distributions over actions. Autoregressive generation supplies all five, and it supplies them so directly that the mapping is almost embarrassing.
action at = yt ∈ V
policy πθ(at | st) = pθ(yt | x, y<t)
transition st+1 = st ⊕ at
trajectory τ = (a1, …, aT) The softmax over the vocabulary is not like a policy. It is a policy: a normalised distribution over a finite action set, conditioned on a state.
Read the transition line again, because it is the single most consequential fact in this article and it is easy to skim past. In a robot, the environment decides what happens after an action, and it may decide stochastically or adversarially. Here, the environment is a string-concatenation operator. It is deterministic, known in closed form, and has no dynamics of its own.
Half of classical RL exists to cope with an environment you cannot see inside: model-based planning, exploration bonuses for unvisited states, off-policy corrections for a world that drifted. None of that applies to a single-turn generation. There is no environment stochasticity to average over; all randomness in the trajectory comes from your own sampling. When people say RL for LLMs is "easy RL," this is what they mean — and when they discover it is hard anyway, the difficulty is coming from somewhere else entirely: the size of the action space, the length of the horizon, and the sparsity of the reward.
The action space, though, is not friendly. A robot arm has maybe seven continuous degrees of freedom; an Atari agent has eighteen buttons. A modern language model chooses from roughly 100,000–260,000 discrete actions at every step, and takes thousands of steps. The state space is the set of all strings. Nothing in that is small.
| Classical RL | Language model | Scale |
|---|---|---|
state st | prompt plus everything generated so far | unbounded, never revisited |
action at | one token from the vocabulary | |V| ≈ 105 |
policy πθ | the model's softmax head | the entire network |
| transition | append the token | deterministic, free |
| episode | one completion, until EOS or a length cap | T = 102–105 |
reward R(τ) | verifier, judge, or reward model score | one scalar, at the end |
discount γ | usually 1 | see note below |
The discount row deserves a sentence. In classical RL, γ < 1 exists partly to keep infinite-horizon returns finite and partly to express genuine impatience. Language episodes are finite and there is no reason to prefer a correct answer at token 40 over a correct answer at token 400, so the standard choice in LLM post-training is γ = 1. This matters more than it sounds: with γ = 1 and a single terminal reward, every token in the sequence has the same return. We will spend §7 and §8 trying to break that tie.
Figure 2 unrolls the whole construction. It starts at a single state, zooms into the softmax to show the policy, follows one trajectory to termination, then pulls back to reveal the branching tree that the policy actually defines.
Reading it. Stages 2–4 hold the camera close on one decision so the policy is legible as a distribution rather than a diagram. Stage 6 pulls back to the full tree; the width of each branch is its probability under πθ. Note in stage 7 that reward lands only at leaves — nothing is scored on the way down.
Stage 6 is the picture to keep. A policy is not a path; it is a weighting over all paths. Training does not move a point through a maze. It reallocates probability mass across an exponentially large tree, using a handful of sampled paths as its only evidence about what is out there.
That framing also tells you where the difficulty lives. With |V| = 105 and T = 103, the tree has 105000 leaves. You will sample eight of them per prompt. Any method that hopes to work must extract a usable gradient from a sample of measure zero — which is exactly what the next two sections construct.
The objective
We have a policy and we have a number that arrives at the end of each episode. What are we maximising? The honest answer is: the average score of the answers this model produces, over the distribution of answers this model produces.
Two features of this expression control everything downstream, and they pull in opposite directions.
θ appears in the measure, not in the integrand. This is the structural fact that separates RL from supervised learning. In a supervised loss, the parameters sit inside the function you are differentiating. Here they sit inside the weights of the average. R knows nothing about θ; it is a black box that takes text and returns a number. It may be a Python interpreter. It may be a person. It has no derivative and we will never ask for one.
The sum is intractable and the samples are yours. You cannot enumerate τ. You can only draw a handful of samples from πθ and average. And because the distribution you are sampling from is the thing you are changing, the estimator's own sampling distribution moves under your feet at every step.
Figure 3 makes the first point concrete. Drag θ and watch: the reward bars never move. Only the weights move. J is the shaded overlap between them.
Reading it. Seven completions, with rewards fixed by an external checker. Moving θ reallocates probability among them; J(θ) is the sum of products. The "greedy trap" button pushes almost all mass onto the single best-known completion — J is high, entropy is near zero, and the policy has stopped being able to discover anything. Keep that picture; it is §9 in miniature.
J is an expectation under your own sampling distribution. A policy that answers correctly 60% of the time with high diversity and one that answers correctly 60% of the time by always emitting the same answer have identical J. They are not equally useful, and they behave very differently under further training — the second one has no gradient signal left to find, because every sample looks the same. This is why entropy shows up as a first-class quantity later, and why "reward went up" is a dangerously incomplete report of an RL run.
The policy gradient, derived
We want ∇θ J. The obstacle is that θ lives in the sampling distribution, so the usual move — differentiate under the expectation — is not available. What follows is the derivation that gets around this. It is four lines of algebra and one identity, and it is worth doing slowly, because every algorithm in the rest of this series is a modification of the object it produces.
The score-function derivation
- J(θ) = Στ πθ(τ) R(τ) Expected reward, written as an explicit sum so we can see where θ is.
- ∇θ J = Στ ∇θ πθ(τ) · R(τ) Linearity. R(τ) has no θ in it — it is a checker, a judge, a test suite — so it passes through the gradient untouched. Notice that we have not asked R to be differentiable, and we never will.
- … but Στ ∇πθ(τ) R(τ) is not an expectation. This is the actual problem. ∇π is a vector of signed numbers that do not sum to one; there is no distribution to sample from, so no Monte Carlo estimate exists. We need to rewrite the sum so a probability reappears in front.
- ∇ log f = ∇f / f ⇒ ∇f = f · ∇ log f The score-function identity, which is nothing more than the chain rule for the logarithm read backwards. It converts an absolute change into a relative change multiplied by the level.
- ∇θ J = Στ πθ(τ) · ∇θ log πθ(τ) · R(τ) Substituting step 4 into step 2. A probability has reappeared in front of the summand, which is exactly what was missing.
- ∇θ J = 𝔼τ ~ πθ [ R(τ) ∇θ log πθ(τ) ] Now it is an expectation under the policy we are already sampling from. We can estimate it by generating text.
- log πθ(τ) = Σt log πθ(at | st) The trajectory probability factorises by the chain rule. In general RL there would also be transition terms p(st+1|st,at), but those carry no θ and would drop out anyway. Here they are identically 1, because appending a token is deterministic.
- ∇θ J = 𝔼τ [ R(τ) · Σt=1T ∇θ log πθ(at | st) ] The policy gradient theorem, in the form we will actually implement. Every term in the inner sum is something a forward pass already computes.
The estimator follows immediately. Draw N completions, score them, and average:
Figure 4 walks the derivation with the manipulated term highlighted at each step, and holds the camera on the score-function identity long enough to see what it is doing.
Reading it. Stage 4 zooms onto the score-function identity, the only non-mechanical move in the derivation. Stage 6 is the moment the sum becomes an expectation; everything before it is unsimulatable and everything after it is a training loop.
What the estimator is actually saying
Strip away the notation and REINFORCE is one instruction: take the log-probability of everything you just said, scale it by how well it went, and step uphill.
The score ∇ log π(at|st) is the direction in parameter space that makes that particular token, at that particular position more likely. It is exactly the gradient a supervised loss would produce if you pretended at were the gold label. So one way to read the whole method:
If R = 1 for everything, you get plain self-imitation and the model drifts toward whatever it already does. If R is negative for a sample, the sign flips and you perform gradient ascent on its negative log-likelihood — actively unlearning that continuation. Supervised learning has no way to say that.
Why the logarithm, rather than the probability itself? Because ∇ log π = ∇π/π divides by the probability of what happened. A rare action that led to a high reward produces a large score, and a common action produces a small one. This is the estimator quietly correcting for the fact that it saw the rare action only because it got lucky: it up-weights the surprise. It is the same normalisation that appears in importance sampling, and it is also — as we will see in §5 — where a great deal of the variance comes from.
What the derivation quietly assumed
Three assumptions are hiding in those eight lines, and each will come back:
Samples come from πθ itself. The expectation in step 6 is under the current policy. The instant you reuse data generated by an older policy — and you will, because generation is expensive — the estimator is biased unless you correct it. That correction is the ratio in §10.
The gradient is unbiased, not low-variance. Unbiasedness is a statement about infinite samples. With eight rollouts and a scalar reward it says almost nothing about the quality of any individual step. §5 is about how bad this is.
Nothing constrains the size of the step. The derivation gives a direction, not a distance. Follow it far enough and the distribution you are sampling from becomes unrecognisable, at which point the gradient you computed no longer describes the model you have. §9 is about that.
Why the estimator is so noisy
REINFORCE is unbiased. That is a claim about the limit of infinitely many rollouts, and it is the weakest useful guarantee in statistics. What matters in practice is the variance of the estimate you actually get from the sixteen or sixty-four completions you can afford, and here the news is bad in three independent ways.
1. Most of your samples contribute nothing
With a binary verifier and a policy that solves 40% of problems, 60% of your rollouts carry R = 0 and therefore contribute the zero vector to the gradient. You paid full price for the forward pass, the KV cache, and the decode — and got no gradient. Worse, if the policy solves 0% or 100% of a prompt's rollouts, every sample is identical and the prompt is completely uninformative. Practical systems filter these out; that is what "dynamic sampling" means when you see it in a recipe.
2. The signal is one number and the parameter is billions
The score ∇ log π(at|st) is a vector with as many components as the model has parameters. The reward is one scalar. A single rollout therefore delivers, at most, one bit or so of information about a direction in a 1010-dimensional space. Compare with a supervised step, which delivers a full target distribution at every one of thousands of positions. The information ratio is not close.
3. Variance grows with sequence length
Here is the one that bites hardest in the long-chain-of-thought era. The per-trajectory term is R(τ) Σt ∇ log π(at|st). The signal — how much R genuinely depends on any given token — does not grow with T. But the sum accumulates T noisy score vectors, so its magnitude grows roughly like √T for weakly correlated terms. Signal-to-noise falls off as 1/√T.
Figure 5 lets you watch all three effects. The left panel scatters individual gradient estimates around the true direction; the right panel plots how the error of the average falls with N.
Reading it. Grey arrows are rollouts that scored zero — they are drawn as dots at the origin, because that is literally their contribution. Green is the running mean; white is the true gradient. Push the pass rate to 0 or 1 and every sample becomes identical, the mean goes to zero, and the prompt teaches nothing. That degenerate case is common enough to have its own name in production recipes.
Error falls as 1/√N, so halving it costs 4× the rollouts — and rollouts are the expensive part of an RL step, typically dominating wall-clock by a wide margin. Buying variance reduction with compute is the worst available exchange rate. Everything in §6–§8 is an attempt to buy it with structure instead: subtract something, predict something, interpolate between estimators. Those are free.
Baselines, and why they are free
Look again at what REINFORCE does with a batch of rollouts scoring 0.7, 0.9, 0.2, 0.8 and 1.0. Every reward is positive, so every log-probability gets pushed up. The 0.2 rollout — the worst of the five — still gets reinforced, just less enthusiastically than the others.
In expectation this is fine, because the probabilities re-normalise and relative pushes are what survive. But as an estimator it is wasteful in a specific way: a large constant is riding along in every term, and constants contribute nothing to the mean while contributing plenty to the variance. Removing it is the cheapest win in all of policy-gradient methods.
Why subtracting is legal
The proof is three lines and it is worth internalising, because the same identity is the reason value functions, group means, and leave-one-out estimates are all admissible.
The baseline is unbiased
- 𝔼τ~π[ b · ∇ log πθ(τ) ] = b Στ πθ(τ) ∇ log πθ(τ)b is a constant, so it comes out of the expectation.
- = b Στ ∇ πθ(τ)The score-function identity again, run in the other direction: π · ∇ log π = ∇π.
- = b ∇ Στ πθ(τ) = b ∇ 1 = 0The sum of a probability distribution is 1, whatever θ is. The gradient of a constant is zero. So subtracting b changes nothing in expectation — it is free.
"Free" is doing real work in that sentence. You get to subtract any quantity you like, as long as it does not depend on the action taken, and the gradient stays exactly correct. The only thing that changes is the variance — and you get to choose b to minimise it.
Reading it. The parabola is estimator variance as a function of b. Its minimum sits at the mean reward when the score magnitudes are independent of the reward — which is the usual working assumption and the reason "subtract the batch mean" is the default everywhere. Drag b far from the minimum and note that the arrow direction of the estimate barely changes while its spread explodes: bias unaffected, variance ruined.
Choosing b
The plain batch mean is the workhorse. It is trivially computable, it requires no extra model, and it makes the sign of each term mean something interpretable: better than average, or worse than average. That reframing is worth pausing on. Once you subtract a mean, you have stopped asking "was this good?" and started asking "was this better than what I usually do?" — which is the question a learning signal should be asking, since the model can only ever move relative to itself.
Two refinements matter. First, a subtle one: if you compute the mean over a batch that includes the sample you are weighting, the baseline correlates with the sample and a small bias creeps in. Using the mean of the other samples — a leave-one-out baseline — removes it. Second, and far more important: b was allowed to depend on anything except the action. In particular it may depend on the state.
That single move — letting the baseline be a learned function of the state — is the doorway into the next section.
Values, advantages, actor–critic
Set γ = 1 and put the entire reward at the last token, which is the standard LLM setup. Then the return from every position is the same number:
The intuition we want is something like: the tokens that mattered were the ones after which things looked better than they did before. Making that precise requires two functions.
Qπ(s,a) = 𝔼τ~π[ G | st = s, at = a ] how good is this position after taking a
Aπ(s,a) = Qπ(s,a) − Vπ(s) how much did a change things
Read V carefully: it is the expected outcome from a prefix, averaging over everything the policy might do next. It is not a judgement of the text so far. A position can have V = 0.9 while the text so far is bland, if the model reliably finishes such positions well.
The advantage is the object we actually want. It is the answer to "did this token help?", stated as a comparison against the policy's own default behaviour at that position. And its most important property is the one that makes it a legal drop-in:
Substituting it in gives the form that every modern algorithm starts from:
Note what changed structurally. The scalar R(τ) that multiplied the whole sum has become a per-token weight At inside it. Credit is no longer uniform. Figure 7 shows what that looks like on a real-length generation.
Reading it. The value curve is what a critic learns to predict: the expected final score from each prefix. The advantage at t is the jump the curve makes when token t is emitted. Most tokens barely move it — punctuation, connectives, restating the question. A few move it a lot. Those are the tokens the update should be about.
Actor and critic
Nobody hands you Vπ. It depends on the policy, which is changing, so it has to be learned alongside it. That gives the two-component structure:
The critic is not part of the model you ship. It exists only to shape the weights on the actor's gradient. For a language model it is usually the same architecture with the language-modelling head swapped for a scalar head — which is to say, another network of comparable size, with its own optimiser state, sitting in HBM for the entire run. We will price that in §12, and in §14 we will see why a lot of recent practice has decided the price is not worth paying.
Asking a network to predict, from a half-finished chain of reasoning, whether the final answer will be right is close to asking it to solve the problem. Value estimates in long-CoT settings are noisy and systematically miscalibrated near the start of a generation, where almost nothing has been committed to. A biased critic does not merely fail to help — it injects a systematic error into every advantage, which is worse than the variance it was brought in to fix. Hold that thought until §8, which is a dial for exactly this trade.
TD, bootstrapping and GAE
We now have a good idea (Aπ) and no way to compute it. The remaining question is how to estimate the advantage from a rollout and an imperfect critic, and the answer turns out to be a continuous dial between two bad options.
The temporal-difference error
The simplest estimate uses the critic's own prediction one step ahead:
If V were exact, 𝔼[δt] = Aπ(st,at) exactly. That is the whole appeal. And it needs only one transition — no waiting for the episode to end, no accumulating a thousand noisy terms.
The catch is the word "if". δt uses V(st+1) as a stand-in for the entire remaining future. That substitution is called bootstrapping — estimating a value from another estimate — and it is where the bias enters. If the critic is systematically optimistic mid-generation, every δ inherits that optimism, and no amount of extra data will wash it out.
The spectrum
Between "one step of reality, then trust the critic" and "all of reality, trust nothing" there is a family of n-step estimators. GAE (Schulman et al., 2015) does not pick one; it takes an exponentially weighted average of all of them, which collapses into a strikingly simple form:
In practice it is computed backwards in one pass over the sequence — Ât = δt + γλ Ât+1 — which is a handful of lines and costs nothing next to the rollout.
Reading it. With a perfect critic the λ = 0 estimate is exact and there is nothing to trade. Add critic error and a real interior optimum appears: low λ now inherits the critic's mistakes, while high λ inherits whatever the rollout had not yet decided. The usual default of λ ≈ 0.95 sits well to the right of the optimum in this idealised frontier, for two reasons worth knowing — real critics on language are worse than the Gaussian noise modelled here, and their errors are persistent rather than redrawn each step, so they do not average out across a batch the way variance does.
GAE was designed for control problems with dense per-step rewards, where δt contains actual new information at every step. In a language rollout with a single terminal reward, all the intermediate rt are zero, so δt = V(st+1) − V(st) — the advantage estimate is entirely a statement about the critic's own curve until the very last step. With γ = λ = 1 the whole telescoping sum collapses to R − V(st), and GAE stops being a spectrum at all. That degeneracy is one honest reason a lot of current practice quietly abandoned the critic; see §14.
Why you cannot take a big step
Everything so far has produced a direction. Nothing has said how far to walk. In supervised learning that question is answered by tuning a learning rate and moving on, because the data does not care. Here it does, for a reason that has no supervised analogue: the gradient you computed describes the policy you sampled from, and one step later you no longer have that policy.
Two things go wrong, and they are different.
Failure one: the distribution collapses
The objective J is maximised by putting all mass on the best completion you have found. Nothing in the derivation penalises that. Take a few large steps toward high-reward samples and the distribution sharpens; sharper sampling produces less diverse rollouts; less diverse rollouts produce advantages that are all nearly equal; equal advantages produce no gradient. The policy has walked into a corner and pulled the door shut behind it.
This shows up in practice as entropy collapse: policy entropy falls monotonically, reward plateaus, and pass@k for k > 1 gets worse even as pass@1 improves — the model has stopped being able to find anything it did not already know. It is one of the most reliably observed failure modes in reasoning RL, and much of the recent algorithmic work in the area is, in one way or another, entropy management.
Failure two: a small step in θ is a large step in π
Parameter distance is a bad proxy for behavioural distance. Softmax logits are exponentiated; a modest change to a logit near a decision boundary can swing a token's probability by an order of magnitude, while an identical change elsewhere does nothing at all. There is no fixed learning rate that is safe in both places.
So measure the step where it matters — in the space of distributions:
TRPO (Schulman et al., 2015) formalised this: maximise a local surrogate of J, subject to a hard constraint that the mean KL stays under some δ. That gives a guarantee of monotone improvement, at the cost of a conjugate-gradient solve against the Fisher information matrix at every update — which is exactly as pleasant as it sounds on a model with ten billion parameters.
Reading it. The right-hand panel shows why parameter distance is the wrong ruler: a sphere of fixed radius in θ maps to a badly stretched region in distribution space, so the same learning rate is timid along one axis and reckless along another. The KL constraint traces the ellipse instead — equal behavioural change in every direction.
There are two KL terms in a typical LLM RL system and they do unrelated jobs. The trust-region KL is to πold, the policy that generated this batch; it is an optimisation device, it is refreshed every iteration, and its only purpose is to keep the update valid. The reference KL is to πref, the frozen SFT model you started from; it never moves, and its purpose is to stop the policy drifting away from fluent, well-behaved language as it chases reward. Setting one to zero has nothing to do with setting the other to zero, and several published recipes now drop the reference KL entirely while keeping the trust region — and vice versa.
PPO, which now feels inevitable
Three pressures have been building. Rollouts are expensive, so we want to take several gradient steps per batch — which immediately makes the data off-policy. Off-policy data needs an importance correction. And the correction must not be allowed to explode, because a large ratio means we have left the region where the estimate is meaningful.
Start with the correction. If the data came from πθold but we are evaluating πθ, importance sampling gives the ratio:
The naive surrogate 𝔼[rt(θ) At] is correct in expectation and catastrophic in practice: for a positive advantage, the objective rewards driving rt as high as it will go, and there is no term anywhere pushing back. PPO's answer is to make the objective simply stop rewarding movement past a certain point:
The min is doing something subtler than "clip the ratio", and it is worth getting exactly right, because the asymmetry is the design.
Reading it. For A > 0 the objective flattens above 1+ε: once you have made a good token sufficiently more likely, there is no further reward for pushing. For A < 0 it flattens below 1−ε: once you have suppressed a bad token enough, stop — but if the ratio somehow rises above 1+ε the penalty keeps growing, so the objective can always pull a bad token back down. That one-sidedness is deliberate.
Reading the four regions
| Case | Ratio | Objective | Gradient | Meaning |
|---|---|---|---|---|
| A > 0 | r ≤ 1+ε | rA | active | keep raising this token's probability |
| A > 0 | r > 1+ε | (1+ε)A | zero | far enough — you are outside the region this estimate describes |
| A < 0 | r ≥ 1−ε | rA | active | keep lowering it |
| A < 0 | r < 1−ε | (1−ε)A | zero | suppressed enough — stop before you destroy the token entirely |
The whole PPO objective, as implemented, is three terms:
The entropy bonus is §9's failure mode written directly into the loss, and it is the term people most often find themselves tuning when a run goes quiet.
Clipping is a heuristic, not a constraint. It does not bound the KL — it bounds the per-token gradient contribution in one direction, and only for tokens whose ratio has already crossed the threshold. A batch can perfectly well take a large aggregate KL step while no individual ratio is clipped, and implementations routinely add an explicit KL penalty or an early-stopping check on measured KL precisely because the clip alone is not enough. Read the clip as "cheap approximate trust region", not "trust region".
Back to tensors
Everything above is now a program. It is worth writing it out in shapes, because the shapes are where the surprises are — and because an RL step looks nothing like the training step in Inside a Training Step, even though the backward pass at the end is identical.
Four things in that listing deserve more than a line.
The logits tensor is the memory event. With B·G = 256 sequences, T = 4096 and |V| = 128{,}000, materialising logits in bf16 is about 268 GB. You do not do that. The gather-and-discard has to be fused or chunked, exactly as in the pretraining loss — and here it happens up to three times per step, once per model that needs log-probabilities.
Masking is not a detail. Prompt tokens were not chosen by the policy, so they contribute no policy gradient. Padding contributes nothing. Tokens after EOS contribute nothing. Every one of these is a place where a wrong mask produces a run that trains, converges to something, and is quietly wrong. This is the single most common category of bug in RL implementations.
The normalisation of the loss is a design choice with teeth. Do you average the per-token losses within each sequence and then average over sequences, or average over all tokens in the batch at once? These differ whenever sequences have different lengths. The first gives every sequence equal weight, so a token in a 100-token answer counts twenty times more than a token in a 2,000-token answer. The second gives every token equal weight. Long-chain-of-thought training is unusually sensitive to this, and at least one widely used recipe treats moving to token-level normalisation as a headline change rather than a footnote.
The step is dominated by generation. The backward pass is a fixed cost you know how to price. The rollout is autoregressive decoding — memory-bandwidth bound, poorly batched at the tail, with a long straggler distribution because some sequences run to the length cap and most do not. In a straightforward synchronous implementation, generation routinely accounts for the large majority of step time.
Reading it. Shapes assume B = 32 prompts, G = 8 samples each, T = 4,096 and |V| = 128,000. The final stage prices a synchronous step; the numbers are illustrative, but the shape of the bar — generation swamping everything else — is what almost every profile looks like before you do something about it.
If generation dominates and the trainer sits idle, the obvious move is to keep the generator running continuously and let the learner consume slightly stale trajectories. That works, and every large-scale system does some version of it — but it is precisely the move that makes the data off-policy, which is why the importance ratio from §10 stops being a formality and starts being load-bearing. The systems decision and the algorithm decision are the same decision.
The four models in the room
A supervised fine-tune holds one model. Classic PPO-style RLHF holds four, and a fifth copy if you generate on a separate inference engine. This is the practical reason RL post-training is a systems problem before it is an algorithms problem.
| Model | Role | State | Memory per parameter |
|---|---|---|---|
| policy πθ | the thing you are training | trainable | params + grads + optimiser ≈ 16 B (bf16 + fp32 Adam) |
| reference πref | anchor for the KL penalty | frozen | params only ≈ 2 B |
| reward model | scores completions | frozen | params only ≈ 2 B — or 0, if the reward is a verifier |
| critic Vφ | estimates returns for advantages | trainable | params + grads + optimiser ≈ 16 B |
| inference engine | generates the rollouts | a copy of the policy | params + KV cache |
The 16 bytes per parameter figure is the same accounting as in Inside a Training Step: bf16 weights, bf16 gradients, and fp32 momentum, variance and master weights. Two trainable models means paying it twice.
Reading it. Bars are weights and optimiser state only — activations, KV cache and fragmentation sit on top. Switch the critic off and the trainable footprint halves; switch the reward model off (which is what a verifier does) and another full copy leaves. Those two toggles are most of the practical case for the algorithms in §14.
None of this is a reason to prefer one algorithm over another on its own. But it does explain a pattern in the literature that would otherwise look like fashion: the components that got removed first were the ones with the worst ratio of usefulness to bytes. The critic costs as much as the policy and, on long chains of reasoning, is the least reliable part of the system. That is a hard combination to defend.
Why RL lives in post-training
Everything so far has treated RL as a technique you apply to a model that already exists. That is a real constraint, not an accident of history, and it is worth being precise about why — partly because the reasons are interesting, and partly because people are now actively attacking them.
The supervision is astonishingly thin
Compare what a gradient step learns from in the two regimes.
In pretraining, every token position is a labelled example. A global batch of four million tokens produces four million targets, each one a full distribution over the vocabulary: not just this token was right but and here is how wrong every alternative was. The cross-entropy loss is dense in a way that is easy to stop noticing.
In RL with a terminal reward, a batch of 256 rollouts produces 256 numbers. If the reward is binary, that is 256 bits. Those 256 bits have to be spread across roughly a million generated tokens by the credit-assignment machinery of §7–§8, and the machinery cannot manufacture information that was never there.
RL (terminal) ≈ 256 targets/step · 1 bit each
That is roughly four orders of magnitude fewer supervised events per step — and the RL step is more expensive, because you had to autoregressively generate the million tokens before you could score them. Whatever the exact accounting, the ratio of information to compute differs by something like five orders of magnitude. You cannot build a world model out of that. The comparison is heuristic rather than a theorem — "bits" here is an upper bound on what the loss could convey, not a measurement of what the model absorbs — but no reasonable tightening of it changes the conclusion.
And from a random initialisation there is no signal at all
This is the sharper obstacle. Policy gradients only work when rollouts differ in reward. From random weights, a language model samples uniform noise; every sample scores zero; the batch mean baseline is zero; every advantage is zero; the gradient is exactly zero. Not noisy — zero. The estimator from §4 is unbiased and completely uninformative, forever, because no amount of sampling from a uniform distribution over a 128,000-token vocabulary will ever stumble onto a correct proof.
RL needs a policy that already succeeds sometimes. The pass rate does not need to be high — one success in sixty-four is plenty — but it must not be zero. That is what the supervised stages buy: they move the policy into a region where its own samples carry information about which direction is better. Pretraining and SFT are not just initialisation, they are what makes the exploration problem finite.
Reading it. The last stage is the one that matters. At 0% competence every rollout scores zero, every advantage is zero and the update is exactly nothing — no amount of patience helps. The probability that a group of G samples contains any signal is 1 − (1 − p)G, which is why increasing G rescues hard prompts and why so many recipes sample generously per prompt.
There is a tidy way to say this. Supervised learning is told what to do and has to generalise. RL has to find what to do and then reinforce it. Finding is exponentially hard in the length of the thing being found, unless something has already narrowed the search. Pretraining narrows the search from "all token sequences" to "fluent, plausible token sequences", which is a reduction so enormous that everything downstream becomes possible.
But the boundary is being pushed
The argument above says RL is information-poor per unit compute and needs a competent starting point. Neither claim forbids RL during pretraining — they just say that naive RL during pretraining is hopeless. Two 2025 lines of work take the constraint seriously and attack it directly, and both do so the same way: by manufacturing a dense reward that exists at every token.
Reinforcement Pre-Training reframes next-token prediction itself as a reasoning task. The model reasons about what comes next, then commits to a prediction, and the reward is verifiable against the corpus: the ground-truth next token is already there. This converts ordinary pretraining data into RL data with a checkable reward at every position, which is exactly the resource that was missing — and it scales with pretraining compute rather than with human annotation.
Reinforcement Learning Pre-training takes a different route to the same place, defining a dense verifier-free reward as the information gain from thinking: how much a sampled chain of thought reduces the surprisal of the next observed token relative to a no-think baseline. Every token position yields a reward signal, and the reported gains on reasoning benchmarks from a small base model are substantial.
Both are early, and neither has displaced the standard recipe. But they do dissolve the folk explanation. The reason RL sits in post-training is not that RL is somehow inappropriate for learning language. It is that the terminal-reward formulation carries almost no information per unit of compute, and that a policy which never succeeds gives you no gradient. Fix those two things and the boundary moves.
A related and unsettled debate: does RL with verifiable rewards expand what a model can do, or does it sharpen a distribution over solutions the base model could already sample? Results cut both ways — RL-trained models reliably win at pass@1 while sometimes losing to their own base model at large k, which is what you would expect from sharpening rather than acquisition. Other work reports genuine gains that survive that test. It is a live question, it depends on the domain and the sampling budget, and anyone telling you it is settled is selling something. L7 is where this gets its own treatment.
Where the textbook bends
PPO was designed for a specific setting: control problems, learning from scratch, dense per-step rewards, a policy that starts out useless and takes large unstable steps. Language model post-training violates almost every one of those assumptions. The algorithm still works, which is a genuine testament to it, but a lot of its machinery is doing less than the derivation implies — and the last few years of practice have been, in large part, a systematic audit of which parts were load-bearing.
Here is that audit. Now that the first-principles derivation is complete, the names can come out.
1. It is barely a sequential decision problem
The reward arrives once, at the end. There is no intermediate reward, no per-step environment dynamics beyond appending a token, and no stochastic transition — the "environment" is deterministic string concatenation. Formally, the whole generation can be regarded as a single action in a bandit problem, with the prompt as the context.
This has consequences. A per-token value function is being asked to predict the outcome of a process it has no partial information about, from a state that is just a prefix. Much of §8's apparatus — bootstrapping, temporal-difference errors, GAE — is machinery for propagating dense intermediate rewards, and there are none. Revisiting this observation is what motivated a return to REINFORCE-style estimators with group baselines, which are simpler, cheaper, and empirically competitive or better.
2. It starts on-policy and mostly stays there
The clipping in §10 exists because in classic deep RL you reuse a batch for many epochs and the policy drifts far from the one that collected it. In RLHF the policy starts from a strong SFT model, the number of inner epochs is usually small, and the KL penalty already restrains movement. The practical consequence is that the clip fires on a small minority of tokens. The ratio is near 1 by construction, not by enforcement.
This is not an argument for removing the clip — it is cheap insurance, and the tokens where it does fire are exactly the pathological ones. It is an argument for not attributing PPO's success in this domain primarily to the clip.
3. The critic was the first thing to go
From §12: the critic doubles the trainable memory footprint. From point 1: it is being asked to do a job that barely exists. And on long chains of reasoning it is estimating the value of a 3,000-token prefix of an argument, which is genuinely hard. Bad usefulness-to-cost, so it went.
The replacement is the observation from §6 that any action-independent baseline is valid. Sample G completions for the same prompt, use their mean reward as the baseline, and you have an unbiased advantage with no learned network at all. This is the core of the group-relative family of methods, and it is where L7 begins. A middle path also exists: estimate values by Monte Carlo rollouts from intermediate states rather than learning a network, trading compute for the critic's bias.
4. The KL term is less standardised than it looks
Everyone writes β KL(πθ ‖ πref) and then does different things. Some implementations subtract a per-token KL from the reward before computing advantages, so the penalty flows through the value function; others add it directly to the loss. The KL itself is a sampled estimate, and there are several estimators with different variance and sign properties — the naive one can go negative, which is alarming to watch and mildly harmful to optimise against. And in verifiable-reward settings, a growing number of recipes drop the KL entirely, on the grounds that the reward is not a hackable learned model and the drift is the point.
5. Length quietly biases everything
The normalisation choice from §11 turns into a real pathology on long-chain-of-thought tasks. Sequence-level averaging under-weights tokens in long generations; combined with a clip that treats each token independently, this interacts with truncation of over-length samples in ways that push the model toward or away from longer answers for reasons that have nothing to do with the reward. Recent recipes treat token-level normalisation, careful handling of truncated sequences, and asymmetric clip bounds as first-class design choices rather than implementation trivia.
6. Your on-policy algorithm is not on-policy
This is the most interesting recent finding, and the one most likely to matter to anyone building the system rather than reading about it.
Rollouts are generated by an inference engine — vLLM, SGLang — optimised for throughput. Gradients are computed by a training framework with a different parallelism strategy, different kernels, different batching, different reduction orders. Given identical weights, these two do not produce identical token probabilities. The differences are tiny per token, and entirely explicable as floating-point non-determinism.
But the importance ratio in §10 is a product over the sequence, and tiny per-token discrepancies compound. Worse, the discrepancy distribution has heavy tails: occasionally a token gets a materially different probability. The result is that a nominally on-policy algorithm is quietly optimising a biased off-policy objective from the very first step — before any deliberate staleness from asynchronous pipelines is introduced. Reported symptoms include silent quality degradation and outright collapse on long-horizon runs.
The fixes are exactly what the theory prescribes once you accept the diagnosis: treat the gap honestly with an importance-sampling correction πtrain / πinfer, truncated to bound the variance — or attack the numerics directly. One striking result reports that simply using FP16 instead of BF16 substantially narrows the gap, because the extra mantissa bits reduce rounding divergence between the two implementations.
Reading it. The per-token differences are genuinely negligible — that is the trap. A log-ratio with standard deviation σ per token accumulates to σ√T over a sequence, and the ratio is its exponential, so the sequence-level ratio spreads geometrically with length. The heavy-tail slider adds the rarer, larger discrepancies that dominate in practice. Nothing here is a bug in either runtime; it is two correct implementations disagreeing in the last few bits, amplified by a product over thousands of terms.
7. Entropy is a resource that gets spent
A policy-gradient update increases the probability of whatever worked. Repeat this and the distribution sharpens; sharpen it enough and the model stops exploring, the pass rate saturates, and performance plateaus with a policy that is confident and stuck. Entropy collapse is one of the most reliably observed failure modes in long RL runs, and its dynamics turn out to be fairly predictable — entropy falls, performance rises, both flatten together. Countermeasures range from the classical entropy bonus (crude, and it fights the objective) to clip bounds that are deliberately asymmetric so that raising the probability of a rare token is not suppressed the way lowering a common one is.
8. The ratio and the reward disagree about granularity
One last structural mismatch. The reward is defined on the sequence. The importance ratio and the clip in §10 are applied per token. Applying a token-level correction to a sequence-level signal is not obviously the right thing to do, and it becomes less obviously right as sequences get longer and as mixture-of-experts routing makes per-token probabilities noisier. Defining the ratio at the sequence level instead — matching the granularity of the reward to the granularity of the correction — is one of the cleaner recent proposals.
Every item above is the same move: take a component that PPO includes for a good reason, check whether that reason applies to language models, and act on the answer. The critic exists to reduce variance given dense rewards — the rewards are terminal, so replace it with a group mean. The clip exists to handle large off-policy drift — the drift is small, so the clip is insurance rather than mechanism. The ratio exists to correct for a policy mismatch — and it turns out there is a mismatch nobody had accounted for, coming from the numerics. This is what it looks like when a field stops treating an algorithm as a black box.
What we have not asked yet
We have a complete machine. A prompt goes in, samples come out, something scores them, the scores become advantages, the advantages become a gradient, and a clipped objective keeps the step from destroying the policy. Every piece was derived rather than asserted.
But the whole construction rests on an assumption we made in §1 and never revisited: that a reward function exists.
Every equation in this piece takes R(τ) as given. For code that compiles or arithmetic that checks, that is honest — the reward is a program, and the program is right. For "is this a good answer", it is not honest at all. There is no function. There are people with preferences, and preferences are not a scalar.
So the next question is the one this article deliberately avoided: where does R come from when nobody can write it down? The answer — fit a reward model to pairwise human comparisons, then optimise it — opens problems that are arguably harder than anything here. The reward model is itself a neural network with its own errors, and §9's warning acquires a second meaning: you are not just moving too far from the reference policy, you are moving into a region where your reward function is wrong. Optimising a learned reward hard enough makes it worse, reliably and measurably. And once you notice that the reward model and the policy are trained on the same preference data, a question presents itself: is the reward model necessary at all?
That is L6: From RLHF to DPO.
A different branch starts from the opposite premise. In mathematics, code and formal logic the reward is writable — a checker, a test suite, a proof assistant. No reward model, no overoptimisation, no human in the loop. That changes the calculus completely and is where the last two years of reasoning-model progress came from, along with the critic-free methods gestured at in §14 and the unresolved question of whether any of it expands capability or merely sharpens it. That is L7: RL for Reasoning.
Both branches use the machinery derived here. Neither is comprehensible without it.
References
Primary sources, grouped by the argument they support. Where a result is recent enough to be contested, I have tried to say so in the text rather than in the citation.
Policy gradients and the foundations
- Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning · Williams, 1992 — the REINFORCE estimator of §4, and the baseline argument of §6.
- Policy Gradient Methods for Reinforcement Learning with Function Approximation · Sutton, McAllester, Singh & Mansour, 2000 — the policy gradient theorem in the form used here.
- Asynchronous Methods for Deep Reinforcement Learning · Mnih et al., 2016 — actor–critic at scale, and the entropy bonus of §14.
- High-Dimensional Continuous Control Using Generalized Advantage Estimation · Schulman et al., 2015 — GAE and the λ dial of §8.
Trust regions and PPO
- Trust Region Policy Optimization · Schulman et al., 2015 — the monotonic improvement bound behind §9.
- Proximal Policy Optimization Algorithms · Schulman et al., 2017 — the clipped surrogate of §10.
RL from human feedback
- Deep Reinforcement Learning from Human Preferences · Christiano et al., 2017.
- Fine-Tuning Language Models from Human Preferences · Ziegler et al., 2019 — the four-model structure of §12 in its original form.
- Learning to Summarize with Human Feedback · Stiennon et al., 2020.
- Training Language Models to Follow Instructions with Human Feedback · Ouyang et al., 2022 — InstructGPT.
- Scaling Laws for Reward Model Overoptimization · Gao, Schulman & Hilton, 2022 — why §9's constraint has a second meaning, picked up in L6.
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog · Jaques et al., 2019 — early KL-control for dialogue.
Where the textbook bends (§14)
- Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs · Ahmadian et al., 2024 — the bandit framing, and the case that much of PPO's machinery is inactive here.
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models · Shao et al., 2024 — the group-relative baseline that replaces the critic.
- VinePPO: Unlocking RL Potential for LLM Reasoning Through Refined Credit Assignment · Kazemnejad et al., 2024 — Monte Carlo values instead of a learned critic.
- REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models · Hu, 2025.
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale · Yu et al., 2025 — token-level loss, decoupled clip bounds, truncation handling.
- Group Sequence Policy Optimization · Zheng et al., 2025 — matching the granularity of the ratio to the granularity of the reward.
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models · Cui et al., 2025 — entropy collapse and its predictability.
- Defeating the Training-Inference Mismatch via FP16 · Yao et al., 2025 — the numerical-precision route to closing the gap in §14.6.
RL beyond post-training (§13)
- Reinforcement Pre-Training · Dong et al., 2025 — next-token prediction as a verifiable reasoning task.
- RLP: Reinforcement as a Pretraining Objective · Hatamizadeh et al., 2025 — a dense, verifier-free information-gain reward.
Reasoning models, for L7
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning · DeepSeek-AI, 2025.
- Kimi k1.5: Scaling Reinforcement Learning with LLMs · Kimi Team, 2025.
- Tülu 3: Pushing Frontiers in Open Language Model Post-Training · Lambert et al., 2024 — an openly documented full post-training pipeline.