How Language Models Learn to Use Tools

The question underneath the question

Ask someone how a language model uses tools and you will usually get an answer about JSON. That is the easy half. The hard half hides inside a question that sounds naive:

How does the model even know which tools exist?

The short answer is the one that reorganises everything else: usually it does not know, and it never knew. Nothing in the weights of a deployed model encodes the fact that your service exposes get_weather. The application tells the model, at inference time, in the prompt. Tool identity is not a property of the network. It is a property of the current context window.

That single fact has consequences all the way down. It explains why a model can call an API that did not exist when it was trained. It explains why "add more tools" eventually becomes a retrieval problem rather than a prompting problem. It explains why the thing we train is not knowledge of any particular tool but a meta-skill: read a description you have never seen, decide whether it applies, and produce a well-formed invocation.

So the interesting engineering question is not "how do I get JSON out of a model." It is: what, exactly, is the capability, and how is it induced? This article works that question from first principles — starting at the decode loop, where a tool call is nothing but a token that happens to mean something to a program running outside the model.

The shape of one turn

Before any of the theory, it helps to have the mechanics in the body. Here is a complete tool-augmented turn, from the moment an application decides which functions to expose, to the moment the model writes a sentence about the weather in Malmö. Watch where the model stops and where the program takes over. Those two boundaries are the whole subject.

Two things in that loop deserve to be stated plainly, because almost every confusion about agents dissolves once they are.

The model never executes anything. It emits a span of text that a program on the other side chooses to interpret as an instruction. If that program did not exist, the span would be inert — a sentence about a function rather than a call to one. The "action" in "language agent" happens entirely outside the network.

The model has no memory of the call it just made. Between the tool call and the continuation there is a second forward pass, and the only thing carrying state across it is the context window itself. Everything the model "remembers" about what it did is text someone wrote back into the prompt.

A tool call is only tokens

Strip away the API surface and a decoder-only language model is a conditional distribution over sequences, factorised left to right:

There is no term in that product for "call a function." There is no branch, no dispatch, no side effect. Sampling from \(p_\theta\) produces tokens, and tokens are all it ever produces. So where does the action come from?

It comes from a convention layered on top. Fix a set of boundary tokens \(\mathcal{B} \subset \mathcal{V}\) — in practice something like </tool_call>, or a chat-template role delimiter. The runtime decodes normally until it emits a token in \(\mathcal{B}\), then stops. Now define three functions that live entirely in the harness, not the model:

\(\textsf{Parse}\) turns the emitted span into a structured call, \(\textsf{Exec}\) runs it, and \(e\) serialises the result back into tokens. One step of an agent is then just an update of the context:

This is worth sitting with, because it is the cleanest statement of what an agent is. An agent is not a model with extra machinery inside it. It is a fold: repeatedly append the model's output and the world's response to a growing string, and run the model again. The model contributes one thing — a distribution over what text comes next — and the loop contributes everything else.

Where the tokens come from

There are two common encodings, and the distinction matters more for training than for use.

Reserved special tokens. The tokeniser gains dedicated ids for the call delimiters. This is clean — the boundary is unambiguous, it cannot appear by accident in ordinary prose, and a single logit carries the "I am about to act" decision. The cost is that you have changed the vocabulary, so the embedding and output matrices need new rows, and those rows start untrained.

Ordinary token sequences. The delimiters are just strings the model learns to produce, like [ and ]. Toolformer deliberately took this route so it would not have to modify the vocabulary of an already-pretrained model at all. It is less tidy but avoids touching the embedding table.

Either way the training signal is the same and it is unromantic: ordinary next-token cross-entropy over a sequence that happens to contain a call. If the target sequence is a call \(a\) written out as tokens, the loss on that span is

and you get to choose which spans are supervised. Masking the tool-result span is standard, since the model did not produce it and should not be trained to predict an API's output. Masking or keeping the reasoning before a call is a live design choice. The point stands: function calling is learnable with plain supervised fine-tuning. No special head, no dispatch module, no architecture change. Which raises the obvious question — if the objective is unchanged, what is actually being learned?

Does the tool live in the weights or the context?

Here is the question sharpened into two competing hypotheses about what a tool-trained model has learned.

The lookup hypothesis. The model has memorised a table \(L\) mapping tool names to usage patterns. Seeing get_weather retrieves the row it stored during training, which tells it the argument names and their order. This works, and it is genuinely part of what happens for very common APIs — a model that saw thousands of requests.get calls in pretraining has an opinion about them independent of any schema you provide.

The interpretation hypothesis. The model has learned a function

which never dereferences the tool's name at all. It reads the description as text, aligns the schema's fields against what the user asked for, and emits the call. The name is a label, not a key.

The difference between them is invisible on any tool the model saw during training, and total on any tool it did not. A lookup has no row for book_train_ticket if book_train_ticket was never in the table; the best it can do is hallucinate the nearest neighbour. An interpreter does not care, because it was never doing lookups.

The training distribution picks the winner

Both hypotheses minimise the same loss. Gradient descent has no preference for the "right" one — it takes whichever is cheaper to represent given the data it sees. So the question becomes a question about the data.

Suppose your fine-tuning set contains \(N\) distinct tools and \(M\) examples. If \(M / N\) is large — each tool appears hundreds of times — memorisation is cheap and available, and there is no gradient pressure to learn \(g\). If \(M/N\) is close to one, memorisation buys almost nothing: a row you see once and never again is not worth the capacity. The only solution that reduces loss across the whole set is the one that reads descriptions.

Tool diversity is not a coverage strategy. It is a regulariser that makes interpretation cheaper than memorisation.

This reframes something that otherwise looks like a benchmark arms race. Look at the API counts in successive tool-learning datasets:

Read naively, that is people chasing a bigger number. Read through the lens above, it is people pushing \(M/N\) down until interpretation is the only strategy that survives. Gorilla makes this explicit in its training design: rather than train the model on a fixed catalogue, it trains with retrieved documentation in the prompt, so that at test time the model adapts when the documentation changes underneath it — a version bump, a renamed argument, a new endpoint. You cannot memorise a moving target, so you had better learn to read.

A caveat worth keeping

The two hypotheses are not exclusive and the truth is a blend. Models do carry priors about common APIs from pretraining, which is why they will confidently produce a plausible-looking stripe.charges.create call with arguments from three versions ago. That prior is useful when it is right and is precisely the mechanism behind hallucinated API calls when it is wrong — the model is reaching for a remembered row instead of reading the schema in front of it. Gorilla frames its whole contribution around exactly this failure: getting arguments right, and not inventing usage the documentation does not support.

Decomposing the decision

"Use a tool" is not one decision. Written out as an action \(a = (\iota, n, \alpha)\) — invoke or not, which name, which arguments — the chain rule gives an exact factorisation:

Each factor is a separate competence with a separate failure mode, and they are not equally hard. Deciding whether to call is a calibration problem: the model must know what it does not know, and know it before it starts answering. Deciding which is a retrieval-and-discrimination problem, hardest when several tools overlap. Deciding arguments is an extraction problem — pull a city, a date, an id out of a conversation and coerce each into the schema's type.

Benchmarks that report a single "function calling accuracy" collapse three different skills into one number. When2Call was built specifically because of this: it evaluates the first factor in isolation — when to call, when to instead ask a follow-up question, and when to admit that none of the available tools can answer — and finds that strong tool-calling models still have real room to improve on it. Abstention is a capability, not the absence of one.

Now multiply

Here is where a small per-step number becomes an alarming end-to-end number. A multi-step task requires every step to be right. With a strict all-or-nothing success criterion over a horizon of \(K\) steps:

Take logs and the geometry becomes obvious. Writing \(\varepsilon = 1-p\) for the per-step error rate,

Failure is exponential in the horizon and linear in the per-step error. Invert it to get the requirement: to hold task success at a target \(\pi\) over \(K\) steps you need

This is the single most useful piece of arithmetic in agent engineering, and it explains a benchmark result that otherwise looks strange. The Berkeley Function Calling Leaderboard reports that current models are strong on single-turn calls while memory, dynamic decision-making and long-horizon reasoning remain open. Those are not separate deficiencies. A model at 95% single-call accuracy is at 60% over ten steps for arithmetic reasons alone, before you add any of the genuinely new difficulties that only appear in multi-step settings.

The escape hatch is recovery, not precision

The product law assumes an error is fatal. That assumption is what makes the numbers so brutal, and it is also the assumption you can attack. Suppose that after a mistake the model detects and repairs it with probability \(q\) per step. The process is no longer a product; it is an absorbing Markov chain with three states — on track, recoverable error, dead — and the survival probability decays far more slowly because the recoverable state feeds back into the healthy one.

The practical reading: at long horizons, adding a percentage point of per-step precision is worth much less than adding the ability to notice a bad tool result and try something else. That is why "failure recovery" appears as a first-class training category in modern tool datasets rather than as an afterthought, and why BFCL's multi-turn suite includes subsets for missing parameters and missing functions — situations where the correct behaviour is to not proceed as if nothing happened.

Before training: prompting a controller

The first tool-using systems trained nothing. They rearranged the prompt.

Why the field moved this way. Routing first separated language from exact computation. Adding a loop let observations correct the model’s assumptions. Prompting those loops did not scale to thousands of tools, so training absorbed the decision of when to call. Once tool catalogs became larger than the context window, schemas had to be retrieved at runtime—turning tool use into a joint retrieval, generation, validation, and recovery problem.

The key transition was from one routing decision to an interleaved reasoning-and-action loop:

Thought 1  I need the year the film was released.
Action 1   search("Kalki 2898 AD")
Obs 1      Indian epic science fiction film released in June 2024...
Thought 2  Now I need the director's previous feature.
Action 2   search("Nag Ashwin filmography")
Obs 2      ...
Thought N  I have both dates. Answer.

Two directions of benefit run through that loop, and ReAct's contribution is that both are present at once. Reasoning improves acting: the thought decides what to search for next and how to handle a surprise. Acting improves reasoning: an observation from the world overwrites whatever the model had assumed, which is exactly the correction that pure chain-of-thought cannot perform, since chain-of-thought is grounded only in the model's own parameters.

ReAct is the conceptual birth of the modern language agent, and it is also the origin of a habit worth questioning. Because the pattern was invented for prompting, it made the reasoning trace a visible part of the output format. When the same pattern is trained into a model rather than prompted, the thought token budget becomes a cost you pay on every step, and the trace is no longer free scaffolding — it is a thing the model can overfit to. Later work found that supervised fine-tuning on distilled deep-thinking traces can produce a model that overthinks and generalises worse, which is one of the arguments for training tool use with reinforcement learning instead. More on that further down.

The limitation of the prompting era was simpler than any of this. Prompting has a context budget and no learning signal. Every capability had to be demonstrated in the window, every time, for every task. That does not scale — not to many tools, not to many steps, and not to a model that should get better at this over time. So the field started training.

Toolformer, or: making a call pay rent

Toolformer asked a question that prompting could not: can a model learn not just how to call a tool, but when — without anyone labelling when?

The trick is a definition of usefulness that requires no human. A tool call is useful if it makes the future easier to predict. That is measurable, it is self-supervised, and it is stated entirely in the currency the model already optimises: nats of cross-entropy.

The criterion, derived

Take a document \(\mathbf{x} = x_1,\dots,x_n\) and a candidate position \(i\). Let \(c_i\) be a proposed call and \(r_i\) its result. Define a weighted loss over the tokens that follow the call site, conditioned on some prefix \(\mathbf{z}\):

Now compare two worlds. In one, the model gets the call and its result:

In the other, it gets neither, or gets the call with the result withheld:

and keep the call only if it earns its keep by at least a threshold:

That min is the most interesting object in the paper, and it is easy to skim past. Without it, the criterion would reward calls for the wrong reason. Writing Calculator(400 / 1400) into a document is itself a strong hint about what comes next — it tells the model that a division is about to be discussed — so a call could reduce loss purely by being a good prompt, with the API's answer contributing nothing. Taking the minimum against the result-withheld condition subtracts that confound. What survives is the value of the information the tool returned, not the value of the gesture of calling it.

The weights are a locality prior

The weighting sequence is not decoration. Toolformer uses

which means a call is credited only for what it does to the next five tokens or so. Without a decay you would credit a call for information that helps two hundred tokens later, which is unfalsifiable — almost any fact helps somewhere downstream. The decay forces the call to land where the information is actually consumed, which is also where you want the model to learn to place it at inference time.

The pipeline, and its bill

The full loop is: prompt the model with a handful of demonstrations to propose calls; keep positions where the probability of starting a call exceeds \(\tau_s\); sample a few candidate calls at each; execute them all; apply the filter; splice survivors back into the original text; fine-tune on the result with the ordinary language-modelling objective.

The elegance is that the augmented corpus contains the same text as the original, only with calls inserted. Fine-tuning on it therefore cannot damage general language ability — and empirically, perplexity with calls disabled is unchanged. You have added a capability without paying for it with the model's other ones.

The bill arrives as sample efficiency. Because the filter is strict and most proposed calls do not pay rent, the yield is low and wildly uneven across tools:

Processing over a million web documents to obtain a few thousand useful calculator examples is not a good exchange rate, and the paper says so. The reason is structural: web text rarely sits at a moment where a division would help the next five tokens, and the heuristics needed to find those moments — documents containing three numbers where one is the product of the other two — are doing a lot of the work.

Two findings that aged well

Tool use is emergent in scale. Applying the same procedure across GPT-2 sizes, the benefit of having tools only appears at around 775M parameters. Below that, models perform about the same with and without calls — not because they cannot emit the syntax, but because they cannot decide when a call would help. The decision is the hard part, and it is the part that needs capacity.

The model is somewhat calibrated about its own ignorance. Under plain greedy decoding, Toolformer calls the API on about 40% of one benchmark's examples, and scores higher on the subset where it chose to call than the no-tool baseline scores overall. It was, to a degree, calling on the questions it would have got wrong. Push the decoding to make calls more eagerly and that calibration disappears — you get more calls and a worse-selected set of them.

A hundred thousand tools

Everything so far assumed the schemas fit in the prompt. Now suppose they do not.

The arithmetic is unforgiving. Let \(N\) be the number of available tools and \(s\) the average tokens per schema — name, description, parameter list, types, required flags. Realistic values put \(s\) somewhere between 80 and 250. Injecting everything costs

which is 40,000 tokens at \(N=400\) before the user has said a word, and roughly \(1.5\times 10^{7}\) at \(N = 10^5\). Even where the context window technically allows it, you are paying prefill on every turn of a multi-step task. And prefill attention is quadratic in sequence length, so the cost of a tool you never call is not zero — it is charged on every step of the loop.

There is a second cost that is easy to miss and often larger. Selection accuracy degrades with the number of distractors. Adding tools does not merely make the prompt longer; it makes the discrimination problem harder, especially since real API catalogues are full of near-duplicates with subtly different constraints — three ways to search flights, two of which do not support one-way trips.

So the schemas have to be filtered before they reach the model. The structure is exactly retrieval-augmented generation with function signatures as documents:

ToolLLM built precisely this: a corpus of 16,464 real RESTful APIs from RapidAPI across 49 categories, paired with a neural API retriever trained to recommend the relevant subset for each instruction, so that nobody has to hand-pick tools at inference time. Gorilla approaches the same problem from the model side with retriever-aware training — the model is trained with retrieved documentation already in its prompt, so it learns to treat that documentation as authoritative rather than as decoration.

The retriever becomes a ceiling

Once discovery is a separate component, the success decomposition gains a term the model cannot influence:

If recall@k is 0.85, no amount of model quality gets you above 0.85. The model cannot call what it cannot see, and worse, it cannot even tell you that it cannot see it — from inside the context, a tool that was not retrieved is indistinguishable from a tool that does not exist. The correct behaviour in that case is abstention, and abstention is the factor models are weakest on.

This produces a genuinely awkward engineering tension. Raise \(k\) and recall improves but discrimination degrades and cost rises. Lower \(k\) and the opposite. There is no setting that is right for all requests, which is why hierarchical schemes — retrieve a category, then tools within it — and multi-stage rerankers keep reappearing.

Search over the call sequence, not just the call

ToolLLM also changed how solution paths are found in the first place. Plain ReAct commits: it takes one action, sees the result, takes the next. A wrong early action poisons everything after it, and there is no mechanism to back out. ToolLLM replaced that with a depth-first search over a decision tree of reasoning traces — expand a node, and if the path dies, abandon it and expand a different one, with the previously generated nodes shown to the model so it produces a genuinely different alternative.

The effect is large. Their reported pass rates put the same fine-tuned model at roughly 66.7% average with tree search against 47.0% with linear ReAct-style decoding. That gap is not a modelling improvement — it is the same weights with a different inference procedure. It is the tool-use instance of a general fact about test-time compute: when a task is a search problem, spending compute on search buys more than spending it on a better single guess.

They also used the search as an annotation procedure, which is the sneakier contribution: run the tree search with a strong model, keep only the paths that succeeded, and you have 126,486 instruction-and-solution-path pairs to fine-tune on. The expensive search happens once, offline, and its results are distilled into a model that can afford to be more direct.

Generalising to tools you never trained on

Return to the question that motivates the whole design. The model never saw book_train_ticket. Why does it work?

Because at inference time the tool is not a name to be resolved. It is a document to be read, and the model has been trained on a task whose structure is: given a document describing a capability and a request describing a need, produce the object that connects them. Nothing in that structure refers to any specific tool.

It is worth being precise about which sub-skills transfer and which do not.

• Argument extraction transfers well. Pulling "Malmö" out of a sentence and binding it to a field named city is a general alignment operation between natural language and a type signature. It has nothing to do with the weather.
• Format compliance transfers well. Emitting a JSON object with the required keys and plausible types is nearly independent of what the keys mean.
• Selection transfers moderately. It depends on descriptions being discriminative. Two tools with vague, overlapping descriptions are confusable no matter how good the model is — this is a documentation problem masquerading as a model problem.
• Semantics transfers poorly. Whether a date should be an ISO string or a Unix timestamp, whether limit is capped at 50, whether calling this endpoint twice is idempotent — these live in prose in the description, if they live anywhere. A model can only honour a constraint that was written down.

That last bullet is the practical takeaway for anyone exposing tools to a model. The gap between a 70%-reliable tool and a 95%-reliable one is very often not in the model but in the description: whether the failure modes are documented, whether the units are stated, whether the required fields are actually marked required. You are writing a prompt, and it is being read by a reader with no ability to ask a colleague what the field means.

Manufacturing the trajectories

Real tool-use logs are scarce, proprietary, and messy. Almost all modern function-calling ability is therefore induced from synthetic data, and the interesting design question is not how to generate it — an LLM will happily generate anything — but how to verify it.

Function calling has a property that most synthetic-data domains lack, and it is the reason this works at all: the answer is executable. A synthetic dialogue about philosophy cannot be checked. A synthetic function call can be parsed, type-checked against a schema, actually run, and its result compared to the intent that produced it. Verification is mechanical for two of the three stages.

The APIGen pattern

APIGen makes this into a pipeline with three hierarchical gates, applied in increasing order of cost:

1. Format check. Does the generated object parse? Are the fields the schema's fields? Are the types right? Cheap, catches most junk.
2. Execution check. Run the call against the real API. Did it succeed? An argument that is well-typed but nonsensical usually dies here.
3. Semantic check. A model judges whether the call, the arguments, the number of calls and the returned results actually match what the query asked for. This is the only stage that requires judgement, and it is applied last, to a set already thinned by the cheap gates.

The released dataset is 60,000 verified entries drawn from 3,673 executable APIs across 21 categories, with human evaluation on a 600-example sample putting correctness above 95%. The headline result is the one that matters for practitioners: models in the 1B–7B range trained on this data reached the top of the function-calling leaderboard at the time, with the 1B model beating substantially larger general-purpose models. On a narrow, verifiable capability, verified data substitutes for scale.

Synthesising the tools themselves

ToolACE takes the idea one step further back. Rather than starting from a fixed catalogue of real APIs, it grows the catalogue: a self-evolution process synthesises 26,507 APIs spanning many domains, data types and constraint patterns, then generates dialogues through interplay between multiple agents, with a complexity evaluator steering difficulty and a two-layer verification system combining rule-based and model-based checks.

Two of its design choices are worth stealing regardless of scale.

Format diversity is deliberate. The same tools are described and called in JSON, YAML, XML and Markdown variants. If every training example uses one serialisation, the model binds the capability to the syntax and becomes brittle when a harness formats things slightly differently. Varying the surface form forces the invariant to be the semantics.

Difficulty is targeted, not maximised. They report that models learn most from data whose complexity sits slightly above current capability — data that is too easy or too hard is unproductive. This is curriculum learning restated, and it argues against the instinct to generate the hardest multi-tool scenarios you can imagine and train on those.

The uncomfortable question hanging over all of this is distributional. Synthetic APIs are generated by a model, so they inherit that model's idea of what an API looks like: tidy names, well-formed descriptions, sensible types. Real enterprise APIs are not like that. They have fields named flag2, undocumented required parameters, and descriptions copied from a ticket in 2019. A model trained exclusively on synthetic tools may have learned to read a genre rather than to read documentation. This is why leaderboards distinguish expert-curated from user-contributed live functions, and why the live categories are harder.

The curriculum

If you were writing the syllabus, what would be on it? Not "call functions" — that is the exam, not the syllabus. The capability decomposes into a set of behaviours that have to be represented in the data, because a behaviour absent from the training distribution is a behaviour the model will improvise.

Read down that column and the boundary moves under your feet. The first four rows are function calling. The last four are agency. There is no line between them — the same model, the same loop, the same loss — only an increasing demand on state tracking. This is exactly the progression the Berkeley Function Calling Leaderboard has followed across its versions: from single-call syntax judged by abstract-syntax-tree matching, to serial and parallel calls, to multi-turn interaction, to holistic agentic evaluation including memory and abstention in stateful settings.

The asymmetry nobody budgets for

Positive examples are easy to generate: pick a tool, invent a query it answers, produce the call. Negative examples are hard, and they carry most of the value.

What does the model need to see in order to learn abstention? It needs conversations where a tool almost applies — where the tool named get_student_record is present, the user asks for a student's grades, and the record schema does not contain grades. Getting that near-miss right requires generating a tool and a request that are semantically adjacent but functionally disjoint, which is a much more delicate generation problem than "make a query this API answers." It is unsurprising that abstention is where models are weakest; it is the behaviour whose training data is hardest to manufacture.

When2Call's response to this is instructive: because the decision is naturally multiple-choice — call, ask, or decline — you can build preference pairs directly, and they found preference optimisation on those pairs improved the behaviour considerably more than ordinary fine-tuning did. That makes sense. Fine-tuning on the correct choice teaches what to do. Preference optimisation on the choice against its alternatives teaches what not to do, and abstention is entirely a not-doing.

Syntax is free, semantics is not

Every serving stack now offers structured outputs: hand it a JSON Schema and the tokens it returns are guaranteed to conform. It is worth understanding exactly what that guarantee is, because it is narrower than it looks and it is not free.

The mechanism

A schema compiles to a grammar, and the grammar compiles to an automaton. At decode step \(t\) the automaton's state determines the set \(\mathcal{A}_t \subset \mathcal{V}\) of tokens that keep the prefix valid. The sampler is then given a masked distribution:

which in an implementation is one line: add \(-\infty\) to the logits of every token outside \(\mathcal{A}_t\), then softmax as usual. The guarantee follows immediately — at every step the prefix stays inside the language, so the final string is in the language. Nothing about the model changed; a mask was applied downstream of it.

The subtlety that explains the empirical results

Here is the part that is rarely stated, and it explains why constrained decoding sometimes makes outputs worse. What you probably want is the model's distribution conditioned on producing a valid object:

What locally-masked decoding actually gives you is the product of the per-step renormalised conditionals. These are not the same distribution. To make them equal you would need, at every step, a factor accounting for how much probability mass of valid continuations remains reachable from the prefix:

Computing \(\alpha\) requires marginalising over all valid futures, which is intractable. So every practical constrained decoder ignores it. The consequence is a systematic bias: prefixes that look locally cheap are over-selected relative to prefixes that lead somewhere good. The decoder can commit to an opening token that is perfectly legal and from which the correct answer is no longer reachable in the grammar — and then it must produce something, so it produces a valid, confident, wrong object.

Constrained decoding does not make the model right. It makes the model parseable, and it can trade some of the former for the latter.

What the measurements say

The empirical picture matches the theory. A study comparing three levels of format restriction — hard constrained decoding, format-restricting instructions, and a two-step natural-language-then-convert approach — found that tighter format restrictions degrade reasoning-heavy tasks, with stricter constraints producing larger drops, while structure helps on classification-style tasks where the answer space is small and the format does most of the work.

The practical recipe that follows is the two-step one, and it is worth adopting as a default: let the model reason without a grammar, then constrain only the final object. The reasoning is where the format pressure does damage; the object is where the guarantee is worth having. This is precisely the shape of a modern tool-calling turn — free-form thinking, then a constrained call — and the fact that the shape was arrived at independently by practitioners and by the analysis above is a good sign.

Two further points, since they bite in production.

Grammars and tool-call formats can collide. If a harness expresses tool calls in one syntax and you activate a JSON Schema grammar for the response, the tool-call tokens may be unreachable under that grammar — the model is structurally prevented from calling anything. The mask does not know that some of the tokens it is banning were the ones that mattered.

The mask enforces the schema and nothing above it. It will guarantee that city is a string. It will not guarantee that the string is a city, that it is the city the user meant, or that the call should have been made. Every constraint that determines whether a call is correct rather than merely well-formed lives outside what any grammar can express.

Rewards you can execute

Supervised fine-tuning on verified trajectories takes you a long way, and for single-call function calling it is close to sufficient. It has a known ceiling, and the ceiling is structural rather than incidental.

Imitation optimises the likelihood of a reference trajectory. But most tool-use tasks have many correct solutions and the reference is one arbitrary member of that set. Worse, imitation never observes the consequences of a mistake, because references contain no mistakes — so the model has no training signal for the single most valuable long-horizon behaviour, which by the argument in section 4 is recovery. And there is evidence that fine-tuning on distilled deep-thinking traces produces overthinking and weaker generalisation, which is the failure mode you would predict from copying the surface form of reasoning rather than its function.

Reinforcement learning fits this domain unusually well for one reason. In most alignment settings the reward is a learned model of human preference, with all the fragility that implies. In tool use, a large part of the reward is computable: did the call parse, did it execute, did the arguments match the reference under abstract-syntax-tree comparison, did the final state of the environment satisfy the goal? Verification is the same machinery the synthetic-data pipelines already built, now moved inside the training loop and applied to the model's own rollouts instead of a teacher's.

ToolRL studies the design of exactly this reward — its type, its granularity, how it should change over training — and reports around a 17% improvement over base models and 15% over supervised fine-tuned ones across function-calling benchmarks, with the headline being that the reward design itself, not the algorithm, carried the gains. Related work such as ReTool focuses the reward on when tools should be invoked during reasoning, optimising for correct outcomes while discouraging unnecessary calls — an explicit push back against the assumption that more tool use is better.

The credit-assignment problem returns

The catch is the usual one, in a harsher form. A trajectory with ten tool calls that ends in failure gives you one bit. Which call was wrong? The trajectory is long, the reward is terminal, and the intermediate steps are unlabelled. This is why so much recent work is about reward granularity rather than reward magnitude: step-level rewards, trajectory-level reward models trained to score tool-use quality, and reward-conditioned variants of group-relative policy optimisation for the multi-turn setting.

Environments have had to become more demanding to match. The τ-bench family evaluates conversational agents in customer-service domains with real tool APIs, policy constraints, and a simulated user; its dual-control successor lets both the agent and the user change the world, which makes the task about gathering missing information and recovering from invalid actions rather than about emitting a correct call. That is a much better proxy for what deployed agents do, and scores on it are correspondingly lower.

If you want the underlying machinery — policy gradients, group-relative advantages, why verifiable rewards behave differently from learned ones — that is the subject of the reinforcement-learning entries in the Understanding LLMs series rather than this article. Here the relevant point is narrower: tool use is one of the few LLM capabilities where the verifier is free, and that shapes everything about how it is trained.

What is still open

It would be easy to end on a tidy progression — prompting, then supervised learning, then reinforcement learning, problem solved. The measurements do not support that, and the gaps are worth naming precisely.

Single calls are close to solved; horizons are not. The leaderboard's own summary is that strong models excel at single-turn calls while memory, dynamic decision-making and long-horizon reasoning remain open. Section 4 predicts most of this from arithmetic, but not all of it — long horizons also mean off-distribution contexts full of the model's own prior outputs, which is a regime no training set covers well.

Abstention is unreliable. Models still call a tool when none applies, and still invent arguments instead of asking for them. This is a calibration problem, and calibration has been stubborn everywhere else in this field too.

We are training mostly on synthetic tools. Whether a model trained on generated API catalogues reads documentation or reads a genre is not settled, and the honest answer is that live, user-contributed function categories are consistently harder than curated ones — which is at least consistent with a distribution gap.

Nobody prices a call. Toolformer noted this in 2023 and it remains largely true: the decision to call is trained as if calls were free. They are not — they cost latency, money, rate limit, and sometimes an irreversible side effect. A model that has never seen a cost signal will not weigh one, and the correct objective is not "call when it helps" but "call when it helps more than it costs."

Irreversibility is unmodelled. Reading is safe; writing is not. The distinction between a query and an action that changes the world is currently enforced in the harness with confirmation dialogs and permission scopes, not in the model. Whether that boundary belongs in the model at all is a genuinely open design question, and the conservative answer — keep the enforcement outside, where you can audit it — is probably right for now.

The thing to keep

If one idea from this article survives, make it the first one. A tool call is a span of tokens that a program outside the model has agreed to interpret. The model does not know your tools; you tell it, every time, in the prompt. What it has been trained to do is read that telling and act on it — and every technique in this article, from a loss differential over five tokens to a retriever over sixteen thousand APIs to a mask on the vocabulary, is a way of making that one reading more reliable.

References

Grouped by topic. Every entry links to its primary source.