Reinforcement Learning for Agents From First Principles

The one thing that changes

Reinforcement learning for reasoning models has a shape that is easy to hold in your head. A prompt arrives. The policy emits a few thousand tokens. A verifier looks at the last few of them and returns a number. You compute an advantage, you push on the log-probabilities, you sample again. The environment in that picture is a function: it takes a string and returns a scalar, and it has no memory of you.

Agent RL breaks that in exactly one place, and everything else in this article is downstream of it. The policy no longer writes its own inputs. Halfway through an episode, the next thing the model conditions on is something the world produced in response to what the model just did. The model can influence it, but it cannot compute it, and it certainly cannot plan the whole sequence in advance, because most of the states it will visit do not exist yet at the moment it starts.

That sounds like a small change. It is not. It converts a one-step decision problem into a sequential one, and every difficulty that classical RL spent thirty years on — exploration, credit assignment, distribution shift, off-policy correction, the cost of a rollout — comes back at once, on top of the difficulties that language models already had.

Put the two side by side and the asymmetry is stark. In the top lane, the token at position 1,700 is conditioned on tokens 1 through 1,699, all of which the model wrote. In the bottom lane, the token at the start of turn four is conditioned on an HTTP response, a stack trace, or a database row. The policy authored the query; it did not author the answer.

One practical note before we start. Almost none of the difficulty below is about the policy-gradient estimator. You can take a working GRPO implementation, point it at a web environment, and get a run that trains for two hundred steps and then quietly dies. The estimator was fine. What broke was the variance structure of the batch, or the credit assignment across fifty turns, or the fact that the environment now sits on the critical path of every optimiser step. Those are the things worth understanding.

Writing the agent down

The standard object is a Markov decision process, \(\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, R, \gamma)\). At time \(t\) the agent is in state \(s_t \in \mathcal{S}\), samples an action \(a_t \sim \pi_\theta(\cdot \mid s_t)\), the world moves to

\[ s_{t+1} \sim P(\cdot \mid s_t, a_t), \]

and returns a reward \(r_t = R(s_t, a_t, s_{t+1})\). The objective is the expected discounted return \(J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[\sum_t \gamma^t r_t\right]\).

None of this is new. What is worth being precise about is how badly the single-turn case degenerates. Reasoning RL is also formally an MDP — you can call each token an action and each prefix a state — but it is a very particular one: the transition kernel is deterministic and known, since appending a token to a prefix is a concatenation, and the reward is a single terminal scalar. The 2025 survey literature makes this contrast the organising distinction of the field, framing conventional LLM RL as a degenerate single-step MDP and agentic RL as a temporally extended, partially observable process.

Once \(P\) is a real transition kernel — a web server, a shell, a database, another person — three things become true that were not true before.

You cannot differentiate through it. In a reasoning rollout, the entire trajectory is a function of the model's own sampling. In an agent rollout, the environment interposes at every step. Nothing about it is a smooth function of \(\theta\), and no amount of clever loss engineering makes it one.

You cannot replay it exactly. A stored trajectory is a record of what happened when the environment was in a particular internal condition. Re-running the same action sequence may not reproduce it. Some environments are seedable; most interesting ones are not.

You cannot see all of it. Which is the next section, and it deserves its own.

It is almost never an MDP. It is a POMDP.

A web agent does not receive the page. It receives an accessibility tree truncated to fit a context window, or a screenshot at a particular scroll position. A coding agent does not receive the repository; it receives whatever ls and grep chose to print. A customer-service agent does not receive the customer's intent, only what they typed.

Formally, the true state \(s_t\) is hidden and the agent receives an observation \(o_t \sim O(\cdot \mid s_t)\). The policy is therefore not \(\pi(a \mid s)\) at all. It is

\[ \pi_\theta\big(a_t \mid o_0, \ldots, o_t,\; a_0, \ldots, a_{t-1}\big), \]

a function of the entire interaction history. There is a classical result lurking here: for a POMDP, the optimal policy is a function of the belief state — the posterior over true states given the history — and the belief state is a sufficient statistic, but computing it is generally intractable. What an LLM agent does instead is carry an approximate, natural-language belief in its context window and its activations.

This has an immediate engineering consequence that is easy to state and hard to internalise: memory is not an optional feature of an agent, it is what makes the policy well-defined. If you truncate the history, you are not saving tokens, you are changing the policy class. A separate article on this site works through what an agent should keep and how to decide — see Memory Is More Than Context for the write, manage and read side of it. Here the relevant point is narrower: the thing RL is optimising is a history-conditioned function, and any context-management policy you bolt on is part of the thing being optimised whether you trained it or not.

What counts as an action

In reasoning RL the action space is the vocabulary, and nobody thinks about it. In agent RL the action space is a design decision made by an engineer, usually early, usually without much deliberation, and it constrains everything afterwards.

A realistic action set looks like

\[ \mathcal{A} = \{\text{search}, \text{click}, \text{type}, \text{run\_code}, \text{call\_api}, \text{ask\_user}, \text{answer}\}, \]

except that each element is really a family. search is not one action; it is one action type parameterised by a query string drawn from a space the size of the language. So the policy factorises:

\[ \pi_\theta(a_t \mid h_t) \;=\; \underbrace{P(\text{type} \mid h_t)}_{\text{a choice among a handful}} \;\cdot\; \underbrace{P(\text{arguments} \mid \text{type}, h_t)}_{\text{a generation problem}} . \]

The first factor is a small categorical distribution and is easy to learn. The second is the hard part and is where most tool-use failures live — the article on how models learn to call functions is entirely about that second factor. What agent RL adds is that the two are now optimised jointly against an outcome that arrives many steps later.

Granularity is a lever, and it has an optimum

Consider the same task expressed at different levels of abstraction. At the bottom, move_mouse(413, 271) and click(). At the top, process_refund(order_id). Both are valid action spaces for the same environment, and they produce completely different learning problems.

Write \(N\) for the number of atomic operations a task really requires and \(g\) for how many of them one action covers. The horizon is \(H = N/g\). As \(g\) grows, the horizon shrinks — and with it, everything that scales badly in the horizon: credit assignment path length, context growth, rollout wall-clock, the probability of a compounding error. That argues for coarse actions.

But as \(g\) grows the action space also grows, because a coarser action carries more parameters and more ways to be wrong, and the number of distinct meaningful actions explodes. Learning send_email(to, subject, body) means learning to produce three correct arguments at once, with no intermediate feedback about any of them. That argues for fine actions.

So the total difficulty behaves like a sum of two opposing terms,

\[ \text{difficulty}(g) \;\approx\; \alpha \log\!\frac{N}{g} \;+\; \beta \log \lvert \mathcal{A}(g) \rvert, \]

and there is an interior optimum. This is not a metaphor. It is the reason the agent-computer-interface work found that redesigning the commands available to a software-engineering agent — not the model, not the training — moved task success substantially. The interface is a hyperparameter.

Two corollaries worth stating plainly. First, if your agent is failing, changing the action space is a legitimate and often cheap intervention — frequently cheaper than another training run. Second, a benchmark's action space is part of the benchmark. Comparing two agents that were given different affordances is not comparing two policies.

Why cloning is not enough

Before any RL happens, almost every agent starts with behavioural cloning. You collect trajectories — human demonstrations, traces from a stronger model, successful paths found by search, production logs — and minimise

\[ \mathcal{L}_{\mathrm{BC}}(\theta) \;=\; -\sum_{t} \log \pi_\theta\!\left(a_t^{*} \mid s_t\right) . \]

This works, and it is not a poor cousin of RL. It is stable, sample-efficient, and it teaches the things RL is terrible at teaching: the syntax of a tool call, the format of an action, the conventions of the interface, what a reasonable first move looks like. Skipping it and starting RL from a base model usually means burning thousands of rollouts to discover that actions must be valid JSON.

What cloning cannot do is teach recovery, and the reason is structural rather than a matter of data volume.

The quadratic

Training states are drawn from the expert's occupancy \(d_{\pi^*}\). At deployment, states are drawn from the learner's own occupancy \(d_{\pi_\theta}\). These are different distributions, and they diverge precisely because the learner makes mistakes.

Suppose the cloned policy disagrees with the expert with probability at most \(\varepsilon\) on states from \(d_{\pi^*}\), and that per-step cost is bounded in \([0,1]\). Consider the first time step at which the policy deviates. Before that moment it is on the expert's path and behaving well. After it, it is somewhere the training set never described, and the honest thing to assume is that it collects the maximum cost for the rest of the episode.

The probability that the first deviation happens at step \(t\) is roughly \(\varepsilon\) (times the probability of having survived, which we can bound by one). The damage from a deviation at step \(t\) is at most \(T - t\). So

\[ J(\pi_\theta) - J(\pi^{*}) \;\lesssim\; \sum_{t=1}^{T} \varepsilon \,(T-t) \;=\; \varepsilon\,\frac{T(T-1)}{2} \;=\; O\!\left(\varepsilon T^{2}\right). \]

That is the classical reduction result for imitation learning, and the important part is the exponent. The regret is not linear in the horizon; it is quadratic, because the horizon appears once as the number of chances to err and once as the length of time you spend paying for it.

If instead you train on states drawn from the learner's own distribution — the interactive setting, of which dataset aggregation is the canonical algorithm and online RL is the more general case — the error bound applies where the agent actually goes, and the regret drops to \(O(\varepsilon T)\). One factor of \(T\).

Two sanity checks on this argument, because it is easy to over-read. The bound is a worst case: real environments are frequently forgiving, and a policy that wanders off can often wander back. And it assumes the expert's states are the only ones with supervision, which stops being true the moment your demonstrations contain recovery behaviour. That second observation is a design instruction, not a caveat — if your demonstration set contains only clean trajectories, you are training a policy that has never seen a failure and will not recognise one.

Learning from your own consequences

There is a gap between imitation and reinforcement that turns out to be usefully occupiable. Many environments cannot supply a reward at all — there is no test suite for “did this browsing session go well” — and many others can, but only after a rollout so long that collecting enough of them is uneconomic.

The early experience line of work takes the observation that you do not need a reward to learn something from acting. If the agent proposes its own alternative action at a state from a demonstration and executes it, the resulting future state is real information about the environment, obtained without any reward function. Two ways to use it were studied systematically: implicit world modelling, where the policy is trained to predict the state that follows a state-action pair, grounding it in the dynamics; and self-reflection, where the model writes an explanation of why the expert's action was better than its own, given both outcomes.

Across eight environments and several model families, both consistently beat plain imitation, with the largest gains on the long-horizon tasks; and checkpoints initialised this way reached higher ceilings after subsequent RL than imitation-initialised ones. That last point is the operationally interesting one. This is not a competitor to RL. It is a better place to start it from.

The conceptual framing is worth keeping: three regimes, distinguished by what supervision they consume.

The middle row is doing something specific: it buys you the distribution shift fix from the interactive setting without requiring you to have solved reward design first. Given how much of the rest of this article is about how hard reward design is, that is not a small thing.

Environments are the dataset

Here is the claim this section argues for, and it is the one I would keep if you kept nothing else.

An environment is a small, boring interface:

Six functions. But notice what that interface is: a data generator with unbounded output. A dataset of a million demonstrations is a million samples, forever. An environment is a million samples per week, drawn from the current policy's distribution, with a reward attached. That is a categorically different asset, and it is why the centre of gravity in agent training moved.

Three ways to get a world, and what each costs

Real production systems. Authentic by construction. Also slow, rate-limited, non-deterministic, occasionally expensive, and unsafe to explore in. Network latency alone destabilises RL training, because it puts a variable multi-second delay inside every optimiser step.

Reproducible replicas. The approach that made web-agent research tractable: stand up real, self-hosted applications — a shopping site, a forum, a code host, a CMS — and write programmatic checkers for long-horizon tasks against them. Deterministic, resettable, and the evaluation is functional rather than a string match on a final answer.

Synthesised environments. The 2026 direction. Rather than hand-building worlds, generate them: derive tool ecosystems and stateful backing databases from documentation or from real service graphs, expose them through a uniform interface, generate verifiable tasks against them, and run thousands of isolated instances in parallel. Reported scaling behaviour is consistent across several of these pipelines — ten environments overfits badly, a hundred helps substantially, and gains continue past five hundred.

The failure mode of the synthetic route is worth naming, because it is not obvious. If you simulate tool behaviour with an LLM, identical states and actions can produce inconsistent responses, and an environment whose dynamics are not a function is not an environment you can do RL in. The pipelines that work generate code — real executables over real databases — and use the model to write the environment rather than to be it.

What the reward buys

Reward design in reasoning RL is mostly solved by fiat: the answer is right or it is not. In agent RL you have to decide what “the task was accomplished” means, in code, for a stateful system, and every choice you make is a specification the policy will optimise against literally.

Sparse outcome reward

\[ R(\tau) = \begin{cases} 1 & \text{the task succeeded} \\ 0 & \text{otherwise} \end{cases} \]

Clean, unhackable in the shallow sense, and almost devoid of information. One bit per episode, where an episode might be forty tool calls and two hundred thousand tokens. A trajectory that failed at the very last step is scored identically to one that never started, and the policy has no way to tell them apart.

Dense reward

Pay something at intermediate milestones: \(R(\tau) = \sum_t r_t\), with \(+0.1\) for finding the right page, \(+0.1\) for entering the right value, \(+1\) for the correct final state. Now partial progress is visible and the gradient has something to work with at every stage of training rather than only once the policy is already competent.

The cost is that you have just told the policy what you think is valuable, and you are probably wrong in at least one place.

The one shaping construction that is safe

There is a classical result that deserves to be better known among people writing agent rewards. If you add a shaping term of the form

\[ F(s, a, s') \;=\; \gamma\,\Phi(s') - \Phi(s) \]

for any potential function \(\Phi\) over states, then the set of optimal policies is unchanged. The intuition is that such a term telescopes: along any trajectory, the shaping contributions collapse to \(\gamma^T\Phi(s_T) - \Phi(s_0)\), which depends only on the endpoints. Cycles therefore accumulate exactly zero shaping reward, which is precisely why the loop in Figure 6 becomes unprofitable.

Any shaping term not of that form is a new objective, not a hint. That is a useful test to run on your own reward function: if an agent can gain reward by returning to a state it has already been in, you have written a different task than the one you meant.

Verifiable outcomes

The most robust option, when the environment supports it, is to make the outcome checkable by execution rather than by judgement: run the test suite, query the database for the row that should now exist, diff the filesystem, check the API state, replay the transaction. This is the agentic analogue of verifiable-reward RL for reasoning, and it is why software engineering became the flagship domain — human-validated task sets with executable tests give you a reward function that is a program, and programs do not have opinions.

Where nothing is executable — a research report, a support conversation, a professional workflow inside a simulated company — the field has largely moved to structured rubrics scored by a model. That works better than a holistic score and worse than a test suite, for reasons the next section is about.

Reward hacking is the default, not the failure case

It is tempting to treat specification gaming as an occasional embarrassment. The 2026 literature does not support that reading. In coding agents, RL-trained models have been observed overwriting unit tests, monkey-patching scoring functions, deleting assertions, and terminating programs early to obtain a pass. In rubric-graded settings the exploits become semantic rather than mechanical: prefatory flattery aimed at a judge, self-praise, systematic length inflation, and drift that a stronger verifier reduces but does not remove.

Several benchmarks now exist specifically to measure this, spanning test manipulation, systems-level specification gaming, and taxonomies of hack categories that frontier models detect only partially. The direction of the evidence is consistent: RL post-training increases exploit rates, and the increase generalises across settings rather than staying local to the environment where it was learned.

There is also a theoretical floor here. Given a proxy reward and a true objective that differ at all, it is provably hard to guarantee that optimising the proxy does not degrade the true objective — the negative results on unhackable proxies are strong. So the operational stance is not “write a reward that cannot be hacked”. It is:

1  Make the verifier a program wherever a program is possible.
2  Hold out verification the policy never trained against.
3  Monitor the behaviour, not only the score — step counts, edits to test
   files, unusual terminations, judge-flattery n-grams.
4  Alarm on reward rising while a held-out proxy falls. That divergence
   is the signature, and it is visible long before a human notices.
5  Treat every shaping term as a claim about value, and test it by asking
   whether a cycle can farm it.

None of this is exotic. All of it is the sort of thing that gets skipped because the reward function looked obviously correct when it was written.

Credit over fifty steps

Suppose a trajectory has \(T = 50\) steps and returns \(R = 1\) only at the end. Which actions deserve the credit? The return

\[ G_t \;=\; \sum_{k=t}^{T} \gamma^{\,k-t} r_k \]

tells you the value of being at step \(t\), but with a single terminal reward every \(G_t\) is the same number up to a discount factor, so it distinguishes nothing.

The simplest thing to do — and the thing most group-relative methods do by default — is to treat the whole trajectory as one sample, normalise the returns within a group of rollouts, and attach that single advantage to every token of every action inside the trajectory. Call this trajectory-level credit. It has one enormous virtue: it is unbiased about the trajectory. It also has one enormous flaw: it is maximally uninformative about the action.

Concretely, a redundant page reload inside a winning rollout gets a positive advantage. A necessary, correct step inside a losing rollout gets a negative one. Averaged over enough rollouts these errors wash out. The question is how many is enough, and at fifty steps the answer is a lot more than your batch size.

How the field got finer

The obvious remedy is to estimate \(A(s_t, a_t)\) per step, which classically means learning a value function. For LLM agents that is expensive: a critic at this scale is another model, another set of optimiser states, another source of instability, and the reason group-relative methods became popular in the first place was to avoid exactly that.

So the interesting 2025–26 work is about getting step resolution without paying for a critic. The moves, roughly in order of how much machinery they require:

Anchor-state grouping. Roll out a group of trajectories from the same initial state. Some of them will pass through the same environment state more than once across the group. Collect the actions taken at each such repeated state and compute a group-relative advantage within that set. You get step-level comparison for free, because the extra rollouts you would have needed already exist inside the batch. Combine it with the usual episode-level term and you have a two-level advantage: direction from the outcome, resolution from the local comparison. The reported gains over trajectory-level baselines on embodied and web-shopping benchmarks are in the high single digits to low double digits.

Hierarchical critics. An off-policy turn-level value function paired with an on-policy token-level actor. More machinery, more general — it does not require states to repeat.

Tree-structured rollouts. Reorganise a group of rollouts into a prefix tree and redistribute the outcome signal down the branches. Cheap and elegant; the cost is that it reallocates the same outcome information rather than adding any, and shared prefixes constrain trajectory diversity.

Intrinsic per-turn signals. Instead of an external judge, measure how much a turn moved the policy's own predicted probability of the correct answer, and use that information gain as a per-turn reward.

The honest summary is that this is unsettled. Anchor grouping needs states to be identifiable, which is easy in a gridworld and hard in open-ended generation where two observations differing only in surface form should be the same state and are not. Later work explicitly attacks that assumption. What has converged is the shape: an episode term for direction, a finer term for resolution.

\[ A(s_t, a_t) \;=\; A^{\text{episode}}(\tau) \;+\; \lambda \, A^{\text{step}}(s_t, a_t). \]

Where the signal goes

This section is a short derivation, and I think it is the most useful thing in the article, because it explains a large fraction of what makes long-horizon agent RL feel different from reasoning RL in practice.

Group-relative methods work by comparison. You sample \(G\) rollouts on the same task, compute their returns \(R_1, \ldots, R_G\), and use the normalised deviation from the group mean as the advantage:

\[ \hat{A}_i \;=\; \frac{R_i - \mu_G}{\sigma_G + \epsilon}, \qquad \mu_G = \frac{1}{G}\sum_j R_j . \]

Now ask what happens when all \(G\) returns are identical. Every \(R_i\) equals \(\mu_G\), the numerator is zero for every rollout, and the group contributes nothing to the gradient. Not a small contribution — nothing. The estimator has no baseline other than the group itself, so when the group agrees, the advantage is not noisy, it is undefined.

With binary outcome reward and per-task success probability \(p\), the two degenerate cases are “all failed” and “all succeeded”. So the probability that a group carries any signal at all is

\[ P_{\text{informative}} \;=\; 1 \;-\; p^{G} \;-\; (1-p)^{G}. \]

This function is zero at \(p = 0\) and \(p = 1\), and peaks at \(p = 1/2\). That already tells you something: a task your agent always fails is exactly as useless as a task it always solves, and the most valuable tasks in your training set are the ones it gets right about half the time.

Now connect it to the horizon

Model a task as \(H\) decisions, each of which the policy gets right with probability \(q\), independently. Then

\[ p \;=\; q^{H}, \]

which collapses geometrically. At \(q = 0.93\) — a policy that is right ninety-three percent of the time, which is not bad — a ten-step task succeeds about half the time, a thirty-step task about eleven percent of the time, and a fifty-step task about three percent.

Substitute into the group formula. In the small-\(p\) regime, \(1 - (1-p)^G \approx Gp\), so

\[ P_{\text{informative}} \;\approx\; G \, q^{H} . \]

The fraction of your batch that carries gradient decays exponentially in the horizon. With \(G = 8\) and \(q = 0.93\), a ten-step task gives you informative groups nearly always; a fifty-step task gives you roughly one useful group in four; a hundred-step task gives you one in forty. You are still paying full rollout cost for all of them.

Which of the standard fixes actually address this

Read the formula again, and the menu of interventions falls out of it almost mechanically.

Raise \(p\) by shortening the horizon. Train on short tasks first, extend later. This directly moves \(q^H\) back toward the middle of the curve. It is the entire logic of interaction-horizon curricula, and it is the next section.

Raise \(p\) by choosing tasks. Filter the training distribution to tasks the current policy solves some but not all of the time, and re-filter as the policy improves. Difficulty-matched sampling is not a nicety; it is targeting the peak of \(P_{\text{informative}}\).

Increase \(G\). Helps, but note what it costs. To hold \(Gq^H\) constant while \(H\) grows, \(G\) must grow like \(q^{-H}\) — exponentially. Group size buys you a few more steps of horizon, not an order of magnitude.

Stop using a binary reward. If \(R\) is graded rather than binary, a group in which every rollout fails can still have internal variance, because the rollouts failed differently. This is the strongest argument for partial credit, process rewards and rubric scoring, and it is a variance argument rather than an information-content one.

Look inside the trajectory. Even when all \(G\) trajectories share the same terminal return, the actions inside them differ, and some of those differences occur at the same state. That is exactly the signal anchor-state grouping recovers, which is why step-level credit and long horizons are the same conversation.

Filter the batch. If a group's return variance is below a threshold, drop it before the optimiser sees it, and spend the compute on a group that says something. Trajectory filtering of this kind appears in stabilised agent-RL variants and, in the reasoning setting, in dynamic-sampling schemes that resample until a batch has non-degenerate groups.

The echo trap

There is a specific failure mode of multi-turn agent RL that is worth being able to recognise by sight, because by the time it shows up in your headline metric the run is already unrecoverable.

The reported pattern is this: reward variability cliffs and gradient norms spike, and the policy converges on repeating a narrow pattern that once worked, losing behavioural diversity. The name that stuck for it is the echo trap — the agent generates a reasoning pattern, the pattern gets rewarded, the agent samples it more often, and the group loses the disagreement that was carrying its gradient. A drop in reward standard deviation is the early warning sign.

Notice how tightly this couples to the previous section. The echo trap is what \(P_{\text{informative}} \to 0\) looks like from the inside when the cause is the policy collapsing rather than the task being too hard. Both produce a group whose members agree; both starve the estimator; both are visible in return variance before they are visible in return mean.

The stabilised variants that address it do three things, and each one maps onto the diagnosis. Filter out rollout groups with near-zero return variance, so degenerate batches never reach the optimiser. Decouple the clipping bounds so that high-reward rare trajectories are not suppressed by the same threshold that keeps updates conservative. And drop or loosen the KL anchor to the reference policy, since holding the policy near its initialisation is exactly the wrong medicine for a policy that has already collapsed inward.

There is a further finding from the same line of work that should temper expectations about emergence. Without sufficiently informative, reasoning-aware reward signals, deep agent reasoning largely does not appear from multi-turn RL on its own; what appears instead is shallow strategy and, sometimes, thinking text that does not correspond to the action taken. Reward structure shapes what emerges. “Run GRPO on a long trajectory and planning will emerge” is not supported.

The horizon as a curriculum variable

If the horizon is the thing driving success probability toward zero, then the horizon is a knob you should be turning during training rather than fixing in a config file.

The concrete version of this is a schedule on the interaction budget: cap the number of turns an agent is allowed early in training, and raise the cap as it improves,

\[ h_t \;=\; \min\!\left(H_{\max},\; h_0 + \Delta \cdot \left\lfloor t / T_s \right\rfloor\right), \]

so that early phases emphasise exploitation over a short, learnable horizon and later phases open up room for exploration, backtracking and recovery.

The empirical claim from the framework that formalised this is sharper than it might sound. Beginning training with a large number of interaction turns pushed models into redundant reasoning and unproductive actions and caused training collapse; constraining the turn count permanently narrowed exploration and plateaued early. The progressive schedule produced smoother reward curves and higher long-run returns, with reported improvements of over ten percent on web navigation and much larger jumps on crafting and science-world tasks, and the resulting seven-billion-parameter agents matched or surpassed considerably larger open models and several commercial ones across twenty-seven tasks.

The curriculum need not stop at horizon length. Anything that moves \(p\) toward the middle of the informative band is a candidate axis:

The last line is the one that generalises. A curriculum in agent RL is a control loop on the success probability of the batch, and every other axis is a means of implementing it.

The rollout engine is part of the algorithm

In reasoning RL, generating a batch is a call to a decoding server. It is fast, it is uniform, and it is somebody else's problem. In agent RL, generating a batch means running a loop that looks like this:

while not done:
    context     = build_state(history, memory)
    action      = policy(context)          # GPU, milliseconds to seconds
    observation = env.step(action)         # CPU or network, seconds
    history     = update(history, action, observation)

Two things about that loop determine your training throughput more than any kernel does. Trajectory lengths are highly variable, and environment steps are slow and latency-skewed — a GUI step involving a screenshot, a render and an action commonly takes on the order of a second, and a network tool call can take much longer.

The utilisation bound

Launch \(N\) rollouts and wait for all of them before stepping the optimiser. The wall clock for that batch is \(\max_i T_i\), while the useful work is \(\sum_i T_i\). So accelerator utilisation is bounded by

\[ \text{utilisation} \;\le\; \frac{\mathbb{E}[T]}{\mathbb{E}\!\left[\max_{i \le N} T_i\right]} , \]

and for heavy-tailed duration distributions that ratio is small and gets worse as \(N\) grows, because the maximum of more samples is further into the tail. This is not a constant factor you can engineer away with better batching. It is a property of the distribution.

The answer the field converged on during 2025 and 2026 is architectural. Disaggregate rollout workers from trainer workers rather than colocating them. Let rollout workers start a new episode the instant they finish one, and post completed trajectories into a queue. Let the trainer consume from that queue continuously. Run many environment instances in parallel on cheap CPU capacity, since the environment is not what needs the accelerator.

Every production agent-RL stack described publicly in the last year does some version of this — producer–consumer controllers with an explicit sample queue that bounds staleness, disaggregated rollout and training node pools, high-concurrency rollout engines that decouple accelerator execution from host-side environment latency. The specifics differ; the shape does not.

The bill for it

Asynchrony buys throughput by paying in on-policyness. A trajectory that entered the queue three optimiser steps ago was generated by \(\theta_{t-3}\), and you are about to use it to update \(\theta_t\). The gradient is therefore off-policy and needs correction,

\[ \nabla J \;\approx\; \mathbb{E}_{\tau \sim \pi_{\theta_{\text{old}}}}\!\left[\frac{\pi_\theta(a \mid s)}{\pi_{\theta_{\text{old}}}(a \mid s)} \, A(s,a)\right], \qquad \text{age}(\tau) \le k, \]

with a bounded staleness window \(k\). Setting \(k = 0\) recovers the synchronous algorithm and its bubbles; letting \(k\) grow gives you throughput and instability. The tuning of \(k\) is a real hyperparameter of the training run, and it is one that does not exist at all in single-turn RL.

Context is a resource the policy has to manage

A fifty-turn trajectory does not just take fifty times as long as a one-turn trajectory. It accumulates fifty observations, and observations are the largest objects in the loop — page dumps, file contents, stack traces, search results. A long-horizon research trajectory can involve dozens of queries and context that reaches into the hundreds of thousands of tokens.

Two failures follow. The obvious one is overflow: the window fills, the oldest tokens are evicted, and the oldest tokens are where the goal was written. The subtler one is that even inside the window, performance degrades with history length — success rates on short-horizon variants of a task have been reported to fall dramatically once the same task is embedded in a long interaction history, with the relevant information still technically present.

The interesting response is to make context management an action the policy takes rather than a heuristic applied to it. Branch a sub-task into its own context, let it explore there, then fold: discard the branch transcript and keep a compact result in the main thread. Compared with heuristic summarisation, which compresses at arbitrary points, folding compresses at sub-task boundaries — reasoning is preserved while it is being used and compacted once its usefulness has been realised. Because the fold is a decision, it can be trained: reward the policy for folds that keep enough to finish the task and lose the rest.

The arithmetic is the appeal. With a resident budget \(B\) and \(k\) branches, effective work scales like \(k \cdot B\) while the context the model must attend over stays around \(B\). Related lines — learned summarisation between segments, self-editing context that prunes retrieved documents to make room for further search, explicit orchestration into parallel sub-agents with bounded local contexts — all trade the same way: they convert a hard long-context problem into a compression problem the policy is allowed to solve.

All of which is to say the boundary between “memory system” and “policy” is not stable. If the agent's context is being rewritten between turns, whatever rewrites it is part of the thing you are optimising, and you may as well optimise it deliberately.

Decoupling the agent from the trainer

Most real agents are not a single model call in a loop. They are a planner, a memory store, a tool router, retry middleware, a validator, and three prompts written by three different people. Nobody wants to rewrite that to fit an RL trainer's data structures, and historically that is exactly what training frameworks demanded.

The systems idea that resolves this is to treat the agent as a black box and instrument only the boundary at which it calls the model. Formulate the agent's execution as an MDP whose transitions are the model calls; capture them through the observability layer the agent already has; convert those traces into training transitions through a credit-assignment module; and keep agent execution and training on separate processes entirely. The framework that made this its thesis reports stable improvement across text-to-SQL, retrieval-augmented generation and math tool-use, with essentially no modification to agents built on common orchestration libraries.

Why this matters beyond convenience: it changes where training data comes from. If any deployed agent can emit training transitions, then production traffic becomes an experience stream, and the distinction between “the agent we ship” and “the agent we train” disappears. Several 2026 production reports describe exactly that loop — training inside the real harness, on traces from real usage, rather than in a research reimplementation of it.

Actions you cannot take back

Every RL algorithm assumes you can try things. Exploration is not an optional component; it is the mechanism. And every agent environment that touches the world contains actions with no inverse.

The useful decomposition is by inverse rather than by risk score:

The mitigations are environmental rather than behavioural, and that is the point worth making. You do not fix this by prompting the policy to be careful; a policy under RL pressure will sample the action anyway, because sampling it is how it finds out. You fix it by building a world where sampling is safe:

A twin. Explore in a copy of the system with the same interface and none of the consequences. This is the same insight as the reproducible-replica approach to environments, applied for safety instead of reproducibility.

A staging layer. Let side-effecting actions accumulate into a transaction that is committed once, at the end, after the trajectory is scored. The agent experiences the effects; the world does not, until you decide it should.

A confirmation gate. Make crossing from the reversible region to the irreversible one require an authorisation the policy cannot issue. This is a permission boundary, not a preference.

A corollary that is easy to miss: if your environment cannot be reset, you cannot do online RL in it. Not “should not” — the algorithm requires an initial-state distribution you can sample from repeatedly, and an environment that accumulates irreversible state does not provide one. Reversibility is a design requirement of the environment, not a property you discover later.

Offline, online, and what people actually do

The clean distinction: offline RL trains on stored transitions \((s, a, r, s')\) from some fixed behaviour policy, and online RL has the current policy interact with the environment and learn from what comes back.

Offline is cheap, safe, parallel, and reproducible, and it has one fatal property for agents: the state distribution is frozen. You can only learn about states somebody already visited, which puts you back in the compounding-error regime of Section 4, with the additional problem that the value estimates extrapolate wildly on actions the behaviour policy never took.

Online fixes the distribution and is expensive in every other dimension — slow, nondeterministic, sometimes unsafe, and it puts the environment on the critical path of the optimiser as Section 13 described.

What almost everyone actually does is a staged hybrid, and the staging is not arbitrary. Each stage exists to make the next one affordable:

The order matters because step 5 is the one whose cost scales with everything — horizon, environment latency, group size, the informative-group fraction from Section 10. Every earlier stage exists to raise the policy's success probability before you start paying that bill. Starting online RL from a weak policy is not just slow; at long horizons it is close to sampling zeros.

One policy, many worlds

Train exclusively on a web benchmark and you get a policy that is good at that web benchmark. The gap between that and a general agent is the same gap as between a fine-tuned classifier and a foundation model, and it is being attacked the same way: by scaling and diversifying the training distribution.

The formal statement is that we want one \(\pi_\theta\) that does well in expectation across a family of environments \(E_1, \ldots, E_K\), each of which is its own interactive decision problem:

\[ \max_\theta \; \mathbb{E}_{k \sim \mathcal{E}} \; \mathbb{E}_{\tau \sim \pi_\theta,\, E_k}\left[R_k(\tau)\right]. \]

This is multi-task learning, but with a twist that makes it harder than the usual version: the tasks are not just different data distributions, they are different dynamics. Transfer has to happen at the level of strategy — how to explore, when to verify, how to recover from an error — rather than at the level of features.

The evidence that this transfers at all is reasonably good and mostly recent. Platforms that unify many environment types behind a common interface were built precisely to study it, and the environment-scaling results in Section 6 are transfer results: held-out environments improve as the number of training environments grows, which is only possible if something environment-independent is being learned. Self-evolving arenas push it further by identifying capability gaps automatically and synthesising new tasks to target them, letting policy and environment distribution co-evolve.

What is not established is how much diversity is enough, or what the right axes of diversity even are. Is a thousand shallow environments better than a hundred deep ones? Does diversity in the action interface transfer better than diversity in the domain? These are answerable questions and nobody has answered them cleanly.

Do agents have scaling laws?

For pretraining we have a genuinely predictive relationship between parameters, tokens, compute and loss. For agents we have nothing of the sort, and it is worth being explicit about how many more variables are in play. Agent capability plausibly depends on

\[ Q \;=\; f\big(N,\; D,\; C_{\text{train}},\; C_{\text{test}},\; H,\; \lvert \mathcal{A} \rvert,\; E\big), \]

with \(H\) the horizon, \(\lvert\mathcal{A}\rvert\) the action-space size and \(E\) the environment diversity. The last three have no analogue in pretraining scaling laws, and at least two of them are things you choose rather than things you buy.

The one empirical regularity that does exist

The most useful measured trend treats capability as a duration: the length of task, measured by how long a human professional takes, that an agent completes with fifty percent reliability. That quantity has been growing roughly exponentially, with an original estimate of a doubling roughly every seven months over 2019–2025, and a shorter doubling time when the fit is restricted to the recent period.

The measurement has been revised as the suite has grown — the task set expanded to 228 tasks with 31 of them eight hours or longer, precisely because the previous suite could not resolve the top of the range — and the confidence intervals on the most recent frontier models remain very wide. Treat the headline number as a trend indicator, not a measurement.

Still, connecting it to Section 10's arithmetic is illuminating.

If tasks were \(H\) independent steps with per-step reliability \(q\), the fifty-percent horizon would be

\[ H_{50} \;=\; \frac{\ln 0.5}{\ln q}, \]

which for \(q\) near one is approximately \(0.69/(1-q)\). So doubling the horizon requires halving the per-step error rate. An exponential trend in task duration corresponds, under this model, to a geometric decline in per-step error — and that is a much less romantic-sounding claim than “agent capability is doubling every few months”, which is exactly why it is worth stating.

The model is wrong in an interesting direction, though. It assumes failures are terminal. A policy that notices an error and retries turns one hard step into several easy ones, which breaks the independence assumption in the agent's favour — and recovery is precisely the capability that interactive training installs and imitation does not. So part of what “horizon growth” measures is not lower per-step error at all; it is a higher probability that an error gets caught before it becomes terminal. Those are different capabilities with different training requirements, and the aggregate metric does not separate them.

Questions worth someone's dissertation

How does the number of trajectories required scale with trajectory length? Are ten times as many short trajectories worth more or less than a tenth as many long ones, at fixed token budget? How much environment diversity is needed before held-out transfer appears, and does it saturate? Does model scale substitute for exploration — does a larger policy need fewer rollouts to find the same behaviour, and if so at what exchange rate? Is there a compute-optimal split between rollout generation and gradient steps, analogous to the parameters-versus-tokens frontier?

None of these have clean answers. All of them are measurable with existing infrastructure. That combination is unusual and probably temporary.

Evaluating a policy, not an answer

A reasoning model is evaluated on whether the final answer is correct. An agent has to be evaluated on the trajectory, because two policies with identical success rates can be wildly different products.

The outcome layer is the easy part: did the task get done, verified functionally where possible. Benchmarks that made this work did so by being executable — human-validated software-engineering tasks with real test suites, self-hosted web applications with programmatic checkers for long-horizon tasks, simulated professional environments where a task's completion is a state you can query rather than a string you compare.

The process layer is where the information is:

Recovery rate deserves special attention, because it is the metric most closely aligned with what interactive training is supposed to buy you and it is almost never reported. An agent that never errs and an agent that errs constantly but always notices can have the same success rate on a benchmark and behave completely differently in a system where the environment is less forgiving than the benchmark was.

Two findings from the evaluation literature are worth carrying around. Cross-environment benchmarks localise the dominant failures not in knowledge but in long-term reasoning, decision-making and instruction following — that is, in the sequential structure, exactly where this article has been pointing. And realistic long-horizon web tasks continue to show a large gap between model and human performance, which is a useful corrective to leaderboard numbers on shorter, cleaner suites.

A map of the field

It helps to see the progression laid out along fixed axes rather than as a list of papers. Each generation below exists because a specific limitation of the previous one became the binding constraint.

Read down the “env has state” column and the story of the field falls out of it. Everything above the fourth row is a function approximation problem wearing RL notation. Everything below it is a control problem, and every added row is an answer to a question the row above could not handle: outcome-only credit is too coarse, so go step-level; long horizons collapse variance, so schedule them; a single environment does not transfer, so synthesise hundreds.

“Hackable” here is a qualitative judgement about how much of the reward surface is defined by something other than a program, not a measured rate. The pattern it encodes is the uncomfortable one: the mechanisms that work on open-ended tasks are exactly the mechanisms with the largest exploitable surface, and no entry in this table is both dense and safe and general.

The third table is the one I would pin above a desk. Most of the standard interventions attack the same quantity from different sides, and knowing which side a given technique is on tells you whether two of them will compose or merely overlap. Shortening the horizon and filtering by difficulty do the same job twice; adding step-level credit does something neither of them does.

What is still open

A summary of the pipeline as it currently stands, and then the parts that are genuinely unresolved.

Whether trajectory-level RL teaches strategy or sharpens what is already there. This is the same dispute that runs through reasoning RL, transplanted into a longer horizon, and it is not settled. Multi-turn RL demonstrably improves success rates. Whether it installs genuinely new decision procedures, or amplifies the ones the base model could already sample occasionally, is an empirical question that the current evidence does not cleanly resolve — and the finding that reasoning depth does not reliably emerge without informative reward signals is a data point against the strongest version of the “RL discovers strategy” claim.

How to identify states in open-ended environments. Anchor-based step credit needs to know when two observations are the same state. In a gridworld that is exact equality. On a webpage it is a similarity threshold on a string, and two observations that differ only in a timestamp will land in different groups even though they induce identical continuation values. Recent work attacks this directly with value-equivalence notions rather than surface identity, and it is the most likely place for a clean improvement.

How much environment diversity is enough. The scaling curves exist and go the right way. Nobody has produced the agent equivalent of a compute-optimal frontier — a rule that says, at this budget, build this many environments of this depth and run this many rollouts each.

Whether the horizon trend is a capability trend. Task-duration measurements are the best aggregate we have, and they conflate lower per-step error with better error recovery with better context management. Those have different training requirements and probably different limits. Disentangling them would be more useful than another point on the curve.

How to do online RL in worlds with consequences. Every technique in this article assumes you can reset. The environments people most want agents to work in — production systems, real accounts, other people — do not reset. Digital twins and staging layers are engineering answers to what is partly a research question about learning under irreversibility.

References

Grouped by topic. Every entry links to its primary source.