The Making of an AI Agent

The loop is the whole difference

Ask a model for the weather in Chennai and it emits a function call, a runtime executes it, the result comes back, the model writes a sentence. That is tool use. It is a genuinely useful capability and it is also, structurally, a lookup: one hop out, one hop back, and the world is exactly as you left it.

Now ask the same model to reconcile last quarter's invoices against the ledger. It has to list files it has not seen, decide which of them matter, open some, discover the schema is not what it assumed, revise, query a database, notice a mismatch, chase the mismatch, and — this is the part that changes everything — write. The moment it writes, the world it is reasoning about is no longer the world it started in. It made that world.

That is the line. Not the number of tools, not the sophistication of the schemas. The line is whether the model's output feeds back into the input through something that has state.

Working definition

An agent is a policy, an environment, an action space, an observation channel, persistent state, and a loop that closes over all five. Remove the loop and you have function calling. Remove the state and you have a batch of independent calls. Remove the environment and you have chain-of-thought.

Everything hard about training agents follows from that closure. The data stops being pairs and becomes sequences. The reward stops being local and becomes terminal. The error stops being a wrong sentence and becomes a wrong world.

Frame 01 / 16

Left: one call, one return, no accumulation. Right: each observation is appended to a context that the next decision is conditioned on, and each action mutates a world that the next observation is drawn from. The right-hand diagram is a Markov decision process wearing a chat transcript as a disguise.

An analogy I keep coming back to: single tool use is a vending machine. You put in a request, you get a thing, the machine is unchanged. An agent is cooking dinner. You taste, you adjust, you burn something, you cannot un-burn it, and every decision from that point on is made in a kitchen that your earlier decisions produced.

The model becomes a policy

If you have read the reinforcement learning posts in this series, the transition here is mostly a change of variable. In RLHF and in reasoning RL we already treated the model as a policy — but a policy over a strange, degenerate MDP.

The token-level MDP is a fiction that works

In ordinary language-model RL, we set the state to the prefix and the action to the next token:

s_t = (x, y_{<t}), \qquad a_t = y_t, \qquad s_{t+1} = s_t \oplus y_t.

This is an MDP, but a peculiar one. The transition function is deterministic, known in closed form, and entirely under the model's control: appending a token is the whole dynamics. There is no environment in any meaningful sense. All the difficulty lives in the reward, which arrives once, at the end.

The agent MDP has a world in it

Now lift the abstraction. An action is no longer a token; it is a complete, well-formed thing you can do:

a_t \in \mathcal{A} = \{\texttt{search}, \texttt{read\_file}, \texttt{write\_file}, \texttt{run\_code}, \texttt{call\_api}, \texttt{ask\_user}, \texttt{answer}\}

with arguments drawn from each tool's schema. The transition is now genuinely external and genuinely unknown:

s_{t+1} \sim T(\cdot \mid s_t, a_t), \qquad o_{t+1} \sim O(\cdot \mid s_{t+1}, a_t).

Two things just became true that were not true before. First, the model does not observe the state — it observes a projection of it. A shell command returns stdout, not the filesystem. A search returns ten snippets, not the web. This is a partially observed MDP, and the model is reasoning about a world it can only ever glimpse through a keyhole.

Second, and more subtly: the thing the policy is actually conditioned on is not the state and not even the observation history. It is the context — a rendered, truncated, reformatted transcript assembled by the runtime:

c_t = \Phi\big(x,\; a_0, o_1,\; a_1, o_2,\; \ldots,\; a_{t-1}, o_t\big), \qquad \pi_\theta(a_t \mid c_t).

In POMDP language, \(c_t\) is the agent's belief state. And here is the part that I think is underappreciated: \(\Phi\) is not learned. Somebody wrote it. Someone decided how tool results are formatted, what gets summarised when the window fills, whether failed calls are retained or dropped. The belief-state constructor of your agent is a piece of application code. When people say "context engineering," this is the object they are engineering.

Frame 01 / 18

The camera starts pressed against individual tokens, where transitions are deterministic appends, then pulls back until each burst of tokens collapses into a single macro-action and the environment's stochastic transitions become visible between them. Same trajectory, two zoom levels. The gradient flows at the token level; the reward is defined at the action level.

The factorisation that matters

An action is emitted as tokens, so the policy factorises twice — once over actions, once over the tokens inside each action:

\pi_\theta(a_t \mid c_t) \;=\; \prod_{j=1}^{m_t} \pi_\theta\!\left(u_{t,j} \,\middle|\, c_t, u_{t,<j}\right)

and the probability of a whole trajectory \(\tau = (c_0, a_0, o_1, a_1, \ldots, a_{H-1}, o_H)\) is

P_\theta(\tau) \;=\; \prod_{t=0}^{H-1} \underbrace{\pi_\theta(a_t \mid c_t)}_{\text{policy}} \cdot \underbrace{P(o_{t+1} \mid s_t, a_t)}_{\text{environment, independent of }\theta}.

Stare at the second factor. It sits inside the sequence — those observation tokens are physically in the context, interleaved with everything the model wrote — but it does not depend on \(\theta\) at all. That single fact is responsible for a surprising amount of practical machinery, and it is where we go next.

Reward

The simplest agent reward is a verifier on the final world state: \(R(\tau) = \mathbf{1}[\,\text{task complete}\,]\). Richer variants add partial credit, cost penalties for wasted calls, or policy-compliance checks. But note what has changed even in the simple case: the verifier no longer reads a string, it reads a database. τ-bench, for instance, scores an episode by comparing the final database state against an annotated goal state rather than by matching the agent's text.

Trajectories are the training data

Instruction tuning trains on pairs. You have \((x, y)\), you maximise \(\log p_\theta(y \mid x)\), and the shape of the data matches the shape of the task exactly.

Agent training does not have that luxury. The unit of data is a trajectory: a task, then an alternating sequence of things the model said and things the world said back, ending in an outcome. Written out, one training example looks like this:

\tau = \big(\,x,\; \underbrace{r_0, a_0}_{\text{model}},\; \underbrace{o_1}_{\text{world}},\; \underbrace{r_1, a_1}_{\text{model}},\; \underbrace{o_2}_{\text{world}},\; \ldots,\; \underbrace{a_{H-1}}_{\text{model}},\; \underbrace{o_H}_{\text{world}}\,\big)

where \(r_t\) is the free-form reasoning the model emits before committing to an action — the "thought" in ReAct's thought–action–observation triple.

You must mask the world's tokens, and the reason is not a heuristic

Here is the mistake that is easy to make and expensive to debug. A trajectory is a single flat token sequence. If you fine-tune on it with a standard next-token objective, you will be computing loss on the observation spans too — on search snippets, on stack traces, on JSON blobs the model never wrote.

Go back to the factorisation. The likelihood we want to maximise is the policy's contribution only:

\mathcal{L}_{\text{SFT}}(\theta) \;=\; -\sum_{t=0}^{H-1} \sum_{j=1}^{m_t} \log \pi_\theta\!\left(u_{t,j} \mid c_t, u_{t,<j}\right), \qquad u \in \text{model-emitted spans only.}

Including observation tokens does not merely add noise. It changes the objective. You would be asking the model to predict what the environment will return — fitting a world model — while calling it policy learning. And because tool outputs are long, repetitive, and high-entropy in ways that have nothing to do with good decision-making, they can dominate the gradient by sheer token count. A trajectory that is 5% decision tokens and 95% retrieved passages will train a retrieval-passage language model.

The same principle reappears in RL, where it has a name. Search-R1 interleaves generation with real retrieval calls and applies retrieved token masking so that the policy-gradient loss is only taken over tokens the model actually produced; the paper reports that models trained with the mask consistently outperform those trained without it.

Frame 01 / 15

A real trajectory is mostly not the agent. As the strip fills in, watch the token budget on the right: the spans that carry the learning signal are a minority of the sequence, and the masked spans are exactly the environment factor that dropped out of the gradient.

The gradient, with the environment removed

For policy-gradient training the same cancellation does the same work. Taking \(\nabla_\theta \log P_\theta(\tau)\), every environment term differentiates to zero and we are left with

\nabla_\theta J(\theta) \;=\; \mathbb{E}_{\tau \sim \pi_\theta}\left[\; R(\tau) \sum_{t=0}^{H-1}\sum_{j=1}^{m_t} \nabla_\theta \log \pi_\theta\!\left(u_{t,j} \mid c_t, u_{t,<j}\right)\right].

This is the standard REINFORCE estimator, and it is worth noticing that we never needed to know \(T\) or \(O\). We never model the environment. We only need to be able to run it. That is the whole reason the frontier moves toward building environments rather than modelling them — a point I will come back to at the end.

It is also worth noticing what this estimator is asking of us. A single scalar \(R(\tau)\) multiplies the gradient of every token in the trajectory. If \(H = 40\) and each action is 200 tokens, one bit of feedback is being spread across eight thousand log-probabilities. Hold that thought.

Where trajectories come from

There are four sources, and in practice every serious system uses at least three of them.

1. Human demonstrations

Someone does the task with the tools and the session is recorded. Highest fidelity, and the only source that captures genuinely human judgement about when to stop, when to ask, when a result smells wrong. Also the only source whose cost scales linearly with the number of trajectories and does not fall over time. Used sparingly, usually to seed and to calibrate.

2. Distillation from a strong teacher, filtered by outcome

Point a capable model at an environment, let it run, keep what worked. This is the workhorse. FireAct showed how much headroom this alone has: fine-tuning Llama2-7B on 500 GPT-4-generated agent trajectories produced a 77% improvement on HotpotQA, and the paper's central positive result is that mixing trajectories from multiple tasks and multiple prompting methods beats any single-source set. Diversity of trajectory shape, not just quantity, is the lever.

AgentTuning made the companion point about catastrophic narrowing. Their AgentInstruct set is deliberately small — 1,866 filtered interaction trajectories across six task families — and it is mixed with ordinary general-domain instruction data. Trained that way, AgentLM-70B reached roughly GPT-3.5-turbo-level performance on held-out agent tasks while keeping its general abilities intact. Train on agent trajectories alone and you get an agent that has forgotten how to be a language model.

Two datasets, one name

Careful in the literature: "AgentInstruct" refers to two unrelated things. THUDM's AgentInstruct (2023) is the 1,866-trajectory agent-tuning set described above. Microsoft's AgentInstruct (2024) is a synthetic-data framework — agentic flows that turn raw documents and code into prompts and responses — used to build a 25M-pair post-training corpus spanning tool use, coding, writing and comprehension, which produced Orca-3.

3. Synthetic environments and synthetic tools

If you cannot find enough real environments, manufacture them. Kimi K2's pipeline is the clearest published description of this at scale: build a tool repository from thousands of real MCP tools scraped from GitHub plus evolved synthetic tool specs, sample tool bundles to define thousands of distinct agents, generate tasks paired with explicit rubrics, then run multi-turn rollouts against a simulated tool executor that maintains persistent state and injects controlled failures. An LLM judge scores each trajectory against its rubric and only passing trajectories survive.

The interesting detail is the sanity check they report: an embedding visualisation showing synthetic tools spanning the same region of tool-space as the real MCP tools. That is the thing you have to verify, because the failure mode of synthetic environments is not that they are fake — it is that they are narrow in a way you do not notice until your agent meets a real API.

4. The model's own exploration, scored by outcome

Let the current policy run, reward what succeeds, update. This is the only source whose data distribution matches the policy being trained, which is exactly why it fixes the failure mode in the next section. It is also the most infrastructure-hungry: you now need the environment live, in the training loop, at rollout throughput.

Source	Cost per trajectory	Coverage of failure modes	Distribution	Scales by
Human demonstration	Very high	Narrow — humans rarely fail on camera	Expert	Headcount
Teacher distillation	Moderate	Narrow — you filter failures out	Teacher	Inference spend
Synthetic environments	Low	Wide, if you inject faults deliberately	Simulator	Environment generators
Self-exploration + RL	Low per sample, high infra	Wide — failures are the signal	On-policy	Rollout throughput

All four feed the same mixture that this series has been assembling since the data post:

\mathcal{D}_{\text{post}} \;=\; \lambda_g \mathcal{D}_{\text{general}} \;+\; \lambda_r \mathcal{D}_{\text{reasoning}} \;+\; \lambda_t \mathcal{D}_{\text{tools}} \;+\; \lambda_a \mathcal{D}_{\text{agents}}

and AgentTuning's result is best read as an empirical statement about \(\lambda_g\): it cannot be zero.

Imitation teaches paths, RL teaches policies

Training signal	On-policy?	Needs expert steps?	Learns recovery?	Credit resolution	Main risk
Behavior cloning	✕	✓	Only if demonstrated	Per token	Compounding distribution shift
Filtered teacher traces	✕	Teacher-generated	Rarely	Per token	Success-only data hides repair
Outcome RL	✓	✕	✓	Whole trajectory	Sparse, ambiguous credit
Process rewards	Usually	Step labels or critic	✓	Per action	Reward hacking and costly labels

Why training moved toward environments. Demonstrations cover successful paths; deployed agents create new states, including failures. On-policy interaction is valuable because it trains on those states, not because reinforcement learning is intrinsically more intelligent.

Suppose your training set contains one successful trajectory for a research task:

\texttt{search} \rightarrow \texttt{open}(\text{result }2) \rightarrow \texttt{extract} \rightarrow \texttt{answer}

Supervised fine-tuning on this teaches the model to reproduce that path. Which is fine, until you notice that there were fifty valid paths, that the second result will not always be the right one, and that the model has been given no way to distinguish "this worked" from "this is what one does."

The compounding-error argument

This is not a vague worry; it has a shape. The classic imitation-learning result is about what happens when a cloned policy makes its first mistake. Suppose your policy matches the expert with error rate at most \(\varepsilon\) on states drawn from the expert's distribution — the distribution your training data came from. Nothing has been promised about states off that distribution, because you have never seen any.

So consider the first divergence. The probability of surviving \(t\) steps without one is \((1-\varepsilon)^t\); the probability that the first mistake lands at step \(t\) is about \(\varepsilon(1-\varepsilon)^{t}\). After it lands you are somewhere your training set says nothing about, and the honest worst-case assumption is that you incur cost on every one of the \(H-t\) remaining steps. Summing:

\mathbb{E}[\text{cost}] \;\lesssim\; \sum_{t=0}^{H-1} \varepsilon(1-\varepsilon)^{t}\,(H-t) \;\approx\; \varepsilon \sum_{t=0}^{H-1}(H - t) \;=\; \varepsilon\,\frac{H(H+1)}{2} \;=\; O(\varepsilon H^2)

for small \(\varepsilon\). That is the Ross–Bagnell result, and the shape of it is the point: behaviour cloning degrades quadratically in horizon, while a method that trains on its own induced state distribution degrades linearly. A 2% per-step error rate is invisible at \(H=3\) and catastrophic at \(H=50\).

Frame 01 / 20

Per-step error 4% Grey band: states the expert demonstrations covered. Once the run leaves it, no guarantee survives.

Drag the error rate and step through the horizon. The corridor is the expert's state distribution; each step the cloned policy has a small chance of stepping outside it, and outside it the failure is absorbing — the world has moved and there is no demonstration for the world it moved to.

Why agents have it worse than the textbook case

The classical analysis assumes you could, in principle, recover — you are in a bad state, but a good policy could still act well from there. Agents violate this in a specific way. A wrong action does not just put you in an unfamiliar state; it can change the world irreversibly. Deleting the wrong row, sending the wrong email, committing a broken migration. The expert distribution is not merely unvisited, it is unreachable.

This is why the recovery behaviours matter so much and why they are so hard to get from clean data. If your trajectories are filtered to successes only — which is what teacher distillation does by construction — the model never sees a single example of noticing a mistake and backing out of it. FireAct's robustness experiment gets at this from another angle: with a search tool degraded to return nothing half the time, a prompted ReAct agent lost about a third of its exact-match score while the fine-tuned agent lost roughly half as much. Exposure to messy trajectories buys robustness that clean ones do not.

What RL adds

RL changes the object being optimised from a path to a distribution over paths:

\pi^\star = \arg\max_\pi \; \mathbb{E}_{\tau \sim \pi}\big[R(\tau)\big]

with the expectation taken under the policy's own rollouts. That single change fixes the distribution mismatch by construction — you are always training on the states you actually visit — and it removes the requirement that someone label intermediate steps.

ARTIST is a clean demonstration of the latter. It trains models with GRPO and outcome-based rewards only, letting the policy decide when, which and how to invoke tools inside multi-turn reasoning chains with no step-level supervision at all, and reports up to 22% absolute improvement over base models, including gains on τ-bench and the harder BFCL v3 subsets. ToolRL comes at the same problem from the reward-design side and reports roughly 17% over base and 15% over SFT models — with the useful negative finding that coarse answer-matching rewards are not enough, because a trajectory can reach the right answer through the wrong tool calls.

A caveat worth stating plainly

It is tempting to compress this into "SFT memorises, RL generalises." Don't. The comparison is confounded almost everywhere it is measured: RL runs see more environment interaction, use different data, and are usually initialised from an SFT checkpoint that did necessary work. The defensible claim is narrower and still important — behaviour cloning trains on a distribution it will not encounter at test time, and on-policy methods do not. That is a statement about the data distribution, not about the moral character of the two loss functions.

Credit assignment over long horizons

Reasoning RL already had a credit-assignment problem: five hundred tokens produced one wrong answer, and which token was to blame? It turned out to be tolerable, largely because the tokens form one continuous argument and errors tend to be locally visible.

Agents make it substantially worse. Consider a failed run:

\texttt{search} \rightarrow \texttt{open} \rightarrow \texttt{read} \rightarrow \texttt{search} \rightarrow \texttt{call\_api} \rightarrow \texttt{run\_code} \rightarrow \texttt{answer} \;\Rightarrow\; R = 0

Maybe the first query was too narrow. Maybe the query was fine and the wrong document was opened. Maybe everything was fine until the code had an off-by-one. All seven actions receive the same \(R = 0\), and REINFORCE will push down the log-probability of the four good ones exactly as hard as the bad one. It is a relay race where only the finish time is recorded.

Frame 01 / 22

Four rollouts from the same task. First the episode-level advantage smears one scalar uniformly across every action in each trajectory. Then the anchor states light up — points where several trajectories genuinely stood in the same place — and the actions taken from them can finally be compared against each other rather than against the average of everything.

The four levers

Discounting and return-to-go. Replace \(R(\tau)\) with \(\sum_{k \ge t} \gamma^{k-t} r_k\), so an action is credited with what came after it rather than with everything. Cheap, and it does something, but with terminal-only reward it mostly just reweights the same scalar.

A learned critic. PPO's baseline \(V_\phi(s_t)\) gives a per-step advantage \(A_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)\). Principled, and expensive here in a way it is not elsewhere: the critic has to read the same enormous, growing agent context, so you are carrying a second large model and paying to embed 50k-token states.

Group-relative advantages. GRPO's trick — sample \(N\) rollouts from the same prompt and normalise within the group — is critic-free and works well in the single-turn case:

A_E(\tau_i) \;=\; \frac{R(\tau_i) - \mu_{\mathcal{G}}}{\sigma_{\mathcal{G}}}, \qquad \mathcal{G} = \{\tau_1, \ldots, \tau_N\}.

But note the granularity. This is one number per trajectory. It tells you that run 3 was better than run 1; it says nothing about which of run 3's twelve actions did the work.

Anchor states. GiGPO's contribution is to recover step-level resolution without a critic, by exploiting something specific to agent environments: trajectories from the same task frequently revisit the same environment state. Group the actions taken from each such repeated state — an anchor state \(\tilde{s}\) — and normalise their discounted returns within that group:

A_S(a_t^{(i)}) \;=\; \frac{R_t^{(i)} - \mu_{G^S(\tilde{s})}}{\sigma_{G^S(\tilde{s})}}, \qquad R_t^{(i)} = \sum_{k=t}^{H}\gamma^{k-t} r_k^{(i)},

then combine the two scales:

A\big(a_t^{(i)}\big) \;=\; A_E(\tau_i) \;+\; \omega\, A_S\big(a_t^{(i)}\big).

The elegance is that the step-level groups are built retroactively from rollouts you already had, so it costs no extra generation and no extra memory. The authors report gains over prompted agents and prior RL methods on ALFWorld and WebShop. The stated limitation is the honest one: it depends on being able to recognise that two states are the same, which gets hard fast in environments where the state is a rendered web page.

The pattern underneath

Every one of these levers is doing the same thing — finding a comparison group. A critic compares an action against a learned expectation. GRPO compares a trajectory against its siblings. GiGPO compares an action against other actions taken from the same place. Credit assignment is, in the end, the search for a fair control group.

Four kinds of state

A chat model has one kind of state: the context window. An agent has at least four, they have different lifetimes, and confusing them is the source of an entire genre of bug.

Model context. What is in the prompt right now. Lifetime: one forward pass. Truncatable, and truncated constantly.
Environment state. The filesystem, the database, the browser session, the calendar. Lifetime: until something changes it. Not in the model. Not recoverable by re-reading the transcript.
Persistent memory. Notes, scratchpads, retrieved summaries the runtime carries between sessions. Lifetime: as long as the store lives.
Tool state. Auth tokens, cursors, rate-limit budgets, open connections. Lifetime: idiosyncratic and usually invisible until it expires mid-run.

The trap is that only the first is visible to the model, and the model has a strong prior that the transcript is the world. Create a calendar event and the world changed; summarise the context to save tokens and the record of that change is gone while the change itself is not. The agent will now happily create it again.

ToolSandbox was built specifically because single-turn, stateless tool benchmarks cannot see any of this. It includes stateful tool execution, implicit dependencies between tools where one tool's effect changes what another tool will accept, a user simulator for on-policy conversation, and milestone-based scoring over arbitrary trajectories. The categories it isolates are instructive precisely because they are the ones that were previously invisible: state dependency, canonicalisation (turning "this Friday" into a date, sometimes needing a tool to do it), and insufficient information, where the necessary tool is deliberately withheld and the correct behaviour is to say so rather than invent an answer. Those three remained hard for the strongest models tested.

Frame 01 / 18

One write action, four state layers. Watch what survives the context compaction at frame 12 and what does not — and then watch the agent, reasoning from a transcript that no longer mentions the write, do it a second time.

The harness: what lives where

If you take one diagram from this article, take this one. The claim it makes is simple and consistently misunderstood: the language model is not the agent. The system is the agent. The model is the policy inside it.

Frame 01 / 24

One turn, traced end to end. The harness assembles a context from tool schemas, memory and instructions; the model emits an action; the harness validates it against the schema, executes it under whatever credentials and sandboxing it owns, catches the error when it fails, and renders the result back into the context. The model participates in exactly two of the eleven stages.

Stated as a boundary: the model chooses, the harness does. And the practical consequence is that a large fraction of "the agent got better" is really "the harness got better" — better tool descriptions, better error surfaces, better context compaction, a smaller and more orthogonal action space. SWE-agent's original insight was exactly this, that purpose-built interfaces beat raw generic ones. WebArena is the negative image of the same point: GPT-4 with chain-of-thought solved 14.41% of its tasks against 78.24% for humans, and the dominant failure was not clumsy action sequences but the agent declaring achievable tasks impossible and giving up.

What is actually in the weights

Now the question I think is the most useful one to hold clearly, because almost every confused conversation about agents is a confusion between these three layers.

Layer	Where it lives	Changes when
Parametric — reading a tool description and knowing what it affords; judging when a tool is worth calling at all; extracting arguments from vague instructions; planning; interpreting an observation; recognising failure and recovering; the general grammar of act-then-look	The weights	You train
In-context — which tools exist right now; their exact names and current parameters; permissions; the user's situation; the state of the world this minute	The prompt	Every turn
Harness — executing the call; credentials; authorisation; schema validation; retries; timeouts; sandboxing; persistence; deciding when to stop	Your code	You deploy

Toolformer is the cleanest evidence that the first layer is real and separable. It learns, self-supervised, to insert API calls into ordinary text — keeping only the calls whose results reduce the model's perplexity on the following tokens — and the resulting 6.7B model outperforms a much larger tool-less GPT-3 on several tasks. Nothing about specific tools was memorised. What was learned was when a tool would help, which is a parametric skill that transfers.

The boundary is not always where people put it. "The model can't use our internal API" is usually an in-context problem (the schema is badly described) or a harness problem (auth fails and the error is swallowed), not a weights problem. Conversely, "the model gives up too easily" or "it never checks its work" are weights problems that no amount of prompt engineering fully fixes.

Where does each capability live?

Weights

learned, transfers

Context

supplied per turn

Harness

your code, never the model

Pick a capability, then choose the layer it belongs to.

Tool retrieval is RAG over actions

Suppose your ecosystem has ten thousand tools. ToolBench collected 16,464 real REST APIs from RapidAPI across 49 categories, so this is not hypothetical.

You cannot put them all in the prompt. If each schema costs \(s\) tokens, the whole catalogue costs \(s|\mathcal{T}|\), which at \(s \approx 150\) and \(|\mathcal{T}| = 10^4\) is 1.5 million tokens — over budget on every axis, and attention cost grows quadratically in sequence length besides. So you retrieve:

q \;\longrightarrow\; R_k(q) = \operatorname*{top-}_{k}\; \operatorname{sim}\big(e(q),\, e(T_i)\big) \;\longrightarrow\; \{T_3, T_{91}, T_{205}\} \;\longrightarrow\; \text{context}

ToolLLM trains exactly this: a neural API retriever that recommends a relevant subset for each instruction, which is then handed to the model for multi-round decision-making. The structural parallel to retrieval-augmented generation is exact, and the difference is the payload:

The parallel

RAG retrieves knowledge to condition what the model says. Tool retrieval retrieves affordances to condition what the model can do. Same embedding index, same top-\(k\), same relevance problem — but a retrieval miss in RAG produces a vaguer answer, while a retrieval miss in tool selection produces an agent that is structurally unable to solve the task and does not know why.

That asymmetry deserves a formula, because it is the tightest constraint in the whole pipeline. If a task genuinely requires the tool set \(\mathcal{T}^\star\), then

P(\text{success}) \;\le\; P\big(\mathcal{T}^\star \subseteq R_k(q)\big).

Retrieval recall is a hard ceiling on task success. No amount of policy improvement gets you above it — and because the agent cannot see what was not retrieved, this failure looks like reasoning failure in every log you will read.

Context budget versus catalogue size

All schemas in prompt: —
Retrieved top-k: —
Success ceiling: —

Catalogue size 10,000 Retrieved k 8 Per-tool recall 94%

The success ceiling assumes a task needs three specific tools and each is retrieved independently at the given recall — a generous assumption, and the ceiling still falls fast. Two nines of per-tool recall is not the same as two nines of task feasibility.

Planning versus execution

Two architectures, and the difference between them is a bet about how stable the world is.

Plan-then-act produces a full plan up front, then executes it. It is legible, easy to review, cheap to run, and it assumes the world you planned against is the world you will execute in.

Interleaved reasoning and acting — ReAct — alternates thought, action, observation, and lets each observation revise the plan. Yao and colleagues framed this as a two-way synergy: reasoning traces let the model induce, track and update plans and handle exceptions, while actions let it gather information the plan needs. On HotpotQA and Fever it reduced the hallucination that pure chain-of-thought suffers from by grounding against a Wikipedia API; on ALFWorld and WebShop it beat imitation and RL baselines by 34 and 10 absolute points with one or two in-context examples.

The reason plans break is not that planning is bad. It is that a plan is a policy computed against a belief state, and every observation is evidence that the belief state was wrong. In a static world you plan once. In a world with other writers in it, a plan has a half-life.

Frame 01 / 18

Both lanes get the same task and the same disruption at step 3 — a file has moved. The upper lane committed to a plan and executes it into the gap. The lower lane pays for an extra observation at every step and buys the ability to notice.

In practice the good systems are hybrid: plan at a coarse grain, act and re-observe at a fine one, and re-plan when the observation contradicts the belief that generated the plan. Which is, if you squint, just model-predictive control with a language model as the model.

Post-training becomes an environment problem

Here is the conclusion the rest of the article has been building toward.

For instruction tuning, the atom of data is a pair:

$$(x, y).$$

You can write one down. You can buy one. You can have a contractor produce ten thousand of them. For agents, the atom is a transition:

(s_t, a_t, r_t, s_{t+1})

and you cannot write one down, because \(s_{t+1}\) is not something you author — it is something that happens. To produce it you need something that executes. The dataset has become a program.

So the bottleneck moves. It is no longer "how do we label more data." It is:

executable tools with realistic failure behaviour, not just realistic schemas;
simulators that maintain state across calls and survive being poked at for fifty steps;
task generators that produce difficulty at the right level and in volume;
verifiers — programmatic where possible, rubric-and-judge where not;
reward functions that resist being gamed by a policy optimising against them;
trajectory filters that keep informative failures instead of discarding everything that is not a success.

The strongest recent evidence that this is the lever comes from the coding-agent side, where the verifier is free. SWE-smith takes any Python codebase, constructs an execution environment for it automatically, and synthesises hundreds to thousands of task instances by breaking existing tests — with execution-based validation, so every task has an objectively checkable solution. That produced 50,000 instances from 128 repositories, an order of magnitude beyond what hand-curation had managed; 5,016 expert trajectories were then collected in those environments and used to fine-tune a 32B model to 40.2% on SWE-bench Verified.

Read that pipeline backwards and the priority ordering is obvious. The model was ordinary. The trajectories were generated, not written. The environments were the artefact.

The shift, in one line

Reasoning RL already made this move: static answer keys gave way to verifiable environments the moment we could check a proof or run a test. Agents complete it. The scarce resource is no longer text that describes a task well — it is a world that will tell you, honestly and cheaply, whether you did it.

Which is also why evaluation and training have collapsed into the same engineering problem. τ-bench scores by diffing the final database against a goal state and measures reliability with \(\text{pass}^k\) — the probability that an agent succeeds on all \(k\) independent trials of the same task — and found frontier function-calling agents solving under half the tasks with \(\text{pass}^8\) below 25% in retail. An environment good enough to measure that is an environment good enough to train in. Increasingly, they are the same repository.

What is still open

I want to be careful not to present any of this as settled. The honest state of things:

Long-horizon credit assignment is unsolved. Anchor-state grouping is a real advance and it depends on state matching, which degrades exactly where you most want it — rich, high-entropy environments where no two states are literally identical. Process reward models help but must themselves be trained, and a process reward model that is wrong is worse than no process reward at all.

We do not know the right mixture. The \(\lambda\) coefficients over general, reasoning, tool and agent data are set empirically per model, and there is no scaling law for them. AgentTuning shows general data cannot be dropped; it does not tell you how much you need.

Synthetic environments have an unmeasured distribution gap. A simulated tool executor is a world model, and it is wrong in ways that are correlated with the way the generator that produced it was wrong. Kimi K2's embedding-coverage check is the right instinct, but coverage in embedding space is not the same as coverage in failure space, and failure behaviour is what robustness is made of.

Reliability is not accuracy. The \(\text{pass}^k\) numbers are the ones that should worry us. An agent that solves a task 65% of the time is not 65% of a useful agent when the task is refunding a customer's money. We do not yet have training objectives that target consistency directly; we optimise expected reward and hope variance follows.

The parametric/harness boundary is moving and nobody has mapped it. Every capability currently in the harness — retries, validation, stopping — is a candidate for absorption into the weights, and every one absorbed makes the system less inspectable. I do not think we have a principled account of which side of that line each capability belongs on. It is mostly being decided by what happens to work.

What I am confident about is the frame. Once a model's output loops back through a world that remembers, you are doing reinforcement learning whether or not you use the words, and the quality of your agent is bounded by the quality of the world you can afford to let it fail in.

References

Grouped by topic. Every entry links to its primary source.

The loop is the whole difference

The model becomes a policy

The token-level MDP is a fiction that works

The agent MDP has a world in it

The factorisation that matters

Trajectories are the training data

You must mask the world's tokens, and the reason is not a heuristic

The gradient, with the environment removed

Where trajectories come from

1. Human demonstrations

2. Distillation from a strong teacher, filtered by outcome

3. Synthetic environments and synthetic tools

4. The model's own exploration, scored by outcome

Imitation teaches paths, RL teaches policies

The compounding-error argument

Why agents have it worse than the textbook case

What RL adds

Credit assignment over long horizons

The four levers

Four kinds of state

The harness: what lives where

What is actually in the weights

Weights

Context

Harness

Tool retrieval is RAG over actions

Planning versus execution

Post-training becomes an environment problem

What is still open

References

Reasoning and acting

Training on agent trajectories

Agentic reinforcement learning

Imitation learning and distribution shift

Environments and evaluation