Inside an AI Agent Harness

The gap between a model and an agent

A language model is a conditional distribution. Feed it a sequence of tokens and it gives you a distribution over what comes next. That is the entire interface. Everything else you have ever seen a model do — apologise, refuse, write a haiku, open a pull request — is that one operation, wrapped.

Write it down honestly and the gap becomes obvious:

\[ \pi_\theta:\; x \longmapsto p(y \mid x) \]

Now list what a coding agent has to do. Inspect a repository. Run pytest. Notice that two tests failed. Remember what it tried three steps ago. Ask before deleting something. Stop after forty turns instead of four hundred. Not one of those is an operation on token probabilities. They are all operations on a world, and the model has no hands.

The program that gives it hands has a name that has quietly become standard: the harness. OpenAI uses the word for the layer underneath Codex CLI, Codex Cloud and the VS Code extension — the part that provides the agent loop and execution logic. Google's new open-source runtime describes itself, in its own README, as "a distributed harness runtime." A year ago this was jargon inside three companies. It is now the noun.

The claim of this piece is narrower and more useful than "the harness matters." It is this:

Which means the quantity people call "agent capability" is not a function of the weights:

\[ Q \;=\; f(\pi_\theta,\; H,\; \mathcal{A},\; \mathcal{E}) \qquad\text{not}\qquad Q \;=\; f(\pi_\theta) \]

where \(H\) is the harness, \(\mathcal{A}\) the action space it exposes, and \(\mathcal{E}\) the environment it wires up. We will spend most of this article on \(H\), because it is the term nobody publishes and everybody varies. At the end we will look at what happens when you hold \(\pi_\theta\) fixed and change only \(H\) — the answer is more dramatic than it has any right to be.

Start by looking at the whole machine at once, because the proportions are the first surprise.

Figure 1. Zooming out from the model to the machine. Each ring is a decision the harness makes on the model's behalf, several times per second, forever.

Three words that keep getting confused

It is worth nailing the vocabulary down, because a lot of arguments about agents are really arguments about which of these three things someone means.

The model is \(\pi_\theta\). It receives tokens and emits tokens. Some of those tokens are, by convention, interpreted as a request to call a tool — but that interpretation is not in the model. It is in the reader.

The harness is the control plane: the program that decides what goes into the context, whether a requested action is permitted, how it gets executed, what comes back, what gets remembered, when to stop, and who pays. It is ordinary software. It has bugs, tests, a release cadence, and — this is the part people miss — a design.

The environment is the thing being operated on: a filesystem, a git repository, a shell, a browser, a CI system, a production database. It is the only part of the system that can tell the agent it was wrong.

So the agent is the composite:

\[ \text{Agent} \;=\; \underbrace{\pi_\theta}_{\text{policy}} \;+\; \underbrace{H}_{\text{harness}} \;+\; \underbrace{\iota}_{\text{environment interface}} \]

The last term is doing more work than it looks. \(\iota\) is what turns "the repository" into "what the model sees of the repository," and turns "run the tests" into a specific subprocess with a specific timeout, in a specific sandbox, whose output is truncated in a specific way. Two harnesses with identical loops and identical tools can differ enormously in \(\iota\) alone.

The loop, written out honestly

Here is the whole agent loop. It is genuinely this small:

context = build_initial_context(user_request)

while True:
    response = model(context)

    if response.is_final:
        return response

    call   = response.tool_call
    result = execute(call)

    context.append(call)
    context.append(result)

OpenAI's published description of the Codex loop has exactly this shape: inference either produces a final assistant message or requests a tool call; the harness runs the tool, appends the output to the input, and queries the model again; the process repeats until the model stops asking for tools. A turn ends with an assistant message, which signals a termination state and returns control to the user.

Nine lines. And now the sentence that this entire article exists to justify:

Before we open those four functions, it helps to see the loop running with the data structures made visible — because one property of the picture below turns out to be worth an enormous amount of money.

Figure 2. One turn, unrolled. The turn is the user-visible unit; the step is the model call. A single turn can contain dozens of steps, and every step re-sends the entire list.

A turn is a fold, not a function call

The loop is a fold over contexts. Writing \(y_k\) for the model's output at step \(k\), \(\mathrm{Parse}\) for the harness's interpretation of it, and \(\mathrm{Exec}\) for the tool execution:

\[ C_{k+1} \;=\; C_k \;\oplus\; y_k \;\oplus\; \varepsilon\big(\mathrm{Exec}(\mathrm{Parse}(y_k))\big) \]

with \(\oplus\) being concatenation and \(\varepsilon\) the encoder that turns a tool result into text the model can read. That last function, \(\varepsilon\), is the one nobody writes down and everybody implements badly. We will give it a section of its own.

Three consequences fall straight out of the fold.

The context is monotone. Nothing is ever removed by the loop itself. If you want removal, you have to build it — and building it is a lossy codec problem, which is §16.

The number of steps is not known in advance. \(K\) is a random variable determined by the model's behaviour on a specific repository. That is a strange property for a production system to have, and it is why budgets are a harness primitive rather than a nicety (§14).

Termination is a syntactic event. The loop exits when \(\mathrm{Parse}(y_k)\) yields no tool call. That is a statement about the model's output format. It is not a statement about whether the software works. The distance between those two things is where a surprising fraction of agent failures live.

Why the naive loop is quadratic

Look again at the fold. At step \(k\) the harness sends \(C_k\), which contains everything from steps \(1 \ldots k-1\). So over a turn of \(K\) steps, the total input the provider must process is

\[ T \;=\; \sum_{k=1}^{K} |C_k| \;=\; \sum_{k=1}^{K}\big(P_0 + k\delta\big) \;=\; K P_0 \;+\; \tfrac{1}{2}\,\delta K(K+1) \;=\; O(K^2) \]

where \(P_0\) is the fixed prefix (instructions, tools, project files) and \(\delta\) the average growth per step. The agent loop is quadratic in its own length. A sixty-step turn does not cost sixty times a one-step turn; it costs closer to two thousand times the marginal step.

This is not a rounding error. With a realistic \(P_0 \approx 14\text{k}\) and \(\delta \approx 2\text{k}\), a sixty-step turn processes about 4.5 million input tokens. OpenAI's engineers put the same observation more bluntly when describing why Codex does not use the stateful previous_response_id parameter: yes, the loop is quadratic in the JSON it sends, and they accept it in exchange for statelessness and Zero Data Retention support, because the network is not where the money goes. Sampling is where the money goes.

Which is exactly why prefix caching is not an optimisation. It is load-bearing architecture.

Figure 3. Interactive. Cumulative billed input tokens across one turn. Drag the cache hit rate from 0 to 100 and watch a parabola become a line.

The straightening

Prefix caching works on exact prefix matches: if the first \(m\) tokens of this request are byte-identical to the first \(m\) tokens of a request the server has already processed, the key-value state for those tokens can be reused instead of recomputed. Suppose a fraction \(h\) of calls hit the cache and cached tokens are billed at a fraction \(r\) of the full rate. Then

\[ T_{\text{eff}} \;=\; \sum_{k=1}^{K}\Big[\, h\big(r\,(P_0 + (k-1)\delta) + \delta\big) \;+\; (1-h)\big(P_0 + k\delta\big) \Big] \]

and at \(h \to 1,\, r \to 0\) this collapses to \(P_0 + K\delta\). Quadratic to linear. The entire saving is contingent on one property: the old prompt must be an exact prefix of the new prompt.

That is a constraint on how you are allowed to write software. It says: you may append to the model's view of the world, and you may not edit it. It rules out the obvious implementation of half a dozen features. And it explains an otherwise baffling engineering choice we will come to in §5, where a harness responds to "the user changed directory" by sending a second message saying so rather than updating the first one.

It is worth pausing on how unusual this is as a software constraint. In most systems, keeping state consistent means updating it in place. Here, updating in place is the expensive operation and duplication is the cheap one. The mental model that works is a ledger: you do not go back and correct entry six, you post a correcting entry at the bottom. Accountants figured this out several centuries ago for reasons that had nothing to do with GPUs, and arrived at the same invariant.

The context builder

Before every single inference, some function assembles the thing the model will see. This is the least glamorous component in the system and the one with the most leverage, because the model cannot act on information the builder did not include, and cannot ignore information the builder did include.

Abstractly the context at step \(k\) is a concatenation of blocks:

\[ C_k \;=\; \big[\, I_{\text{sys}},\; T,\; I_{\text{model}},\; I_{\text{perm}},\; I_{\text{dev}},\; I_{\text{proj}},\; S,\; E,\; U,\; H_k \,\big] \]

with system instructions, tool schemas, model-specific instructions, the permissions description, developer config, project instructions, skill metadata, environment state, the user's message, and the running history. Codex publishes its version of this list, and the ordering is instructive: the API server controls the position of the first three items and the content of the system message; the client controls the tools and the instructions; and then the harness inserts a developer message describing the sandbox, the aggregated project instructions from AGENTS.md files walked from the git root down to the working directory, skill metadata, an <environment_context> block with the current working directory and shell, and finally — last — the thing the user actually typed.

Figure 4. The first prompt, assembled block by block. Block sizes are representative of a mid-sized repository with a few MCP servers attached, not measurements of any specific run.

Two things are worth noticing in that picture.

The first is the proportion. The user's request is a rounding error. Everything else is the harness talking to the model about how to behave, what it may touch, where it is standing, and what tools exist. If you have ever wondered why an agent "ignored" a short instruction, look at the ratio: you appended forty tokens to nine thousand.

The second is the specificity gradient. Project instructions are aggregated from the broadest scope to the narrowest — repository root first, working directory last — because later text wins ties in practice. That is a design choice made by a harness author about how conflicting instructions should resolve, and the model has no say in it.

Repository understanding is partly a harness problem

A fifty-million-token monorepo does not fit in a context window and never will. So the builder has to answer a question that no amount of model capability removes: what code should the model see right now?

There are four families of answer, and real harnesses mix them:

• Model-directed reads. Let the model issue sed -n '40,80p' file.py. Simple, and the model pays a round trip for every guess.
• Search. Give it grep/ripgrep with shaped output. Cheap, and it privileges literal token matches over semantic ones.
• Structural index. Parse the repository and retrieve definitions and references. Aider's repository map is the clean example: a compact overview of the important files with their classes, functions and signatures, so the model has a skeleton of a large repository without receiving every source file.
• Embedding retrieval. Standard RAG, which works less well here than elsewhere for a reason worth stating plainly.

That reason: in code, relevance is not primarily semantic. It is structural. The file you need to see is the one that defines the symbol, or the one that calls it, or the one that was edited in the same commit — none of which is well captured by cosine similarity between an issue description and a source file. A useful relevance score for repositories looks more like

\[ R(f) \;=\; \alpha R_{\text{sem}}(f) \;+\; \beta R_{\text{callgraph}}(f) \;+\; \gamma R_{\text{imports}}(f) \;+\; \lambda R_{\text{recency}}(f) \]

where the last three terms come from a parser and a git log rather than from a vector database, and where \(\beta\) and \(\gamma\) usually deserve to dominate. Retrieval in a repository is a graph problem wearing a search problem's clothes.

The append-only invariant

Now we can pay off §3. If cache hits require an exact prefix match, then every mid-turn mutation to something the model has already seen is a cache miss on everything after it. Formally: if a request's prefix diverges at token position \(i\), the recomputation cost is \(|C| - i\). Editing item six of a forty-item list costs you thirty-four items of GPU work.

The consequence is a design rule that reads like superstition until you see the arithmetic:

So when the sandbox configuration or approval mode changes mid-conversation, Codex inserts a fresh developer message in the same format as the original permissions block rather than rewriting the original. When the working directory changes, it appends a new <environment_context> rather than updating the first one. The model briefly sees two contradictory statements about where it is standing, resolves them by recency, and the cache survives. OpenAI's own account of the GPT‑5.6 harness generalises this into a stated principle: all model-visible history is treated as append-only, tools are presented in a deterministic order, and runtime settings such as approval policies are applied at execution time rather than baked into tool definitions.

Figure 5. Two lanes, one difference. Lane B's "harmless" in-place edit invalidates every cached token after it.

The landmines are not exotic. Changing the model mid-turn changes the model-specific instructions near the front of the prompt. Adding or removing an MCP server changes the tool list. And MCP servers can announce a changed tool list on the fly via a list_changed notification — honouring it mid-conversation is correct behaviour and an expensive cache miss. Codex shipped a bug early on where MCP tools were not enumerated in a consistent order, which was enough on its own to break caching.

Sit with that for a second. A non-deterministic iteration order over a hash map — the kind of thing that would never surface in a code review — became a measurable cost regression, because in this architecture byte-for-byte determinism of the prompt prefix is a performance contract. That is a genuinely new class of bug, and it comes from the harness, not the model.

The agent–computer interface

Humans do not interact with computers through raw system calls. We were given IDEs, terminals, file managers, debuggers, autocomplete, syntax highlighting and undo — an entire industry of affordances built around the specific strengths and failure modes of human cognition.

SWE-agent's central argument, made in 2024 and still the cleanest statement of the idea, is that language-model agents are a new category of end user with their own strengths and failure modes, and that they benefit from interfaces designed for them rather than inherited from us. They called it the agent–computer interface, and the analogy is exact:

\[ \text{GUI} : \text{human} \;\;::\;\; \text{ACI} : \text{agent} \]

The Linux shell is a human interface. It looks agent-friendly — text in, text out, turn-based — which is precisely why it is such a seductive default. But it was designed for someone who can scroll, who has a screen with persistent state, who can tell at a glance that a command silently succeeded, and who will notice a stray character in a heredoc. An agent has none of those properties. It has a fixed-width window, no persistence except what you paste back, and an unfortunate willingness to believe that a command it never verified did what it intended.

SWE-agent's numbers made the point stick: with a purpose-built file viewer, search interface, editor with syntax checking, and shaped observations, the same base model solved 12.5% of SWE-bench where the previous best retrieval-augmented system managed 3.8%. Their ablation on a 300-issue subset attributed 10.7 percentage points to the interface alone, relative to an agent given nothing but the default Linux shell. The interface also transferred to a different model than the one it was tuned on, which is the evidence that it was an interface effect rather than a prompt-fitting effect.

The design principles they extracted are worth restating because they are not obvious:

• Actions should be simple to emit correctly. Prefer a small command with two arguments over a shell incantation with six flags, even when the incantation is more powerful.
• Observations should be concise and informative. A search that returns fifty matches with three lines of context each is worse than one that returns "17 matches in 4 files" plus the file list.
• Guardrails belong in the interface. An editor that syntax-checks the result and refuses to write a file it just broke removes an entire failure mode from the model's shoulders.

That third one is the deepest, and it generalises far beyond editing. We will come back to it in §11.

Choosing an action space

Say we want the model to change some code. What are we willing to accept as an action?

Option A: give it one tool, shell(command). The action space \(\mathcal{A}_{\text{low}}\) is enormous — anything expressible in bash, which is everything. Option B: give it read_file(path, start, end), search(query), apply_patch(diff), run_tests(scope). The action space \(\mathcal{A}_{\text{high}}\) is a tiny subset of the first.

Same model. Very different agent. And the reason is not mysterious: we changed what the policy is choosing among.

Figure 6. One task, three action spaces. The turn counts are illustrative of the mechanism; the measured deltas underneath are from the cited studies.

Why code beats JSON on hard tasks

There is a third option that deserves its own treatment: let the action be a program. The CodeAct line of work consolidated agent actions into executable Python, and found up to a 20.7-point absolute improvement in success rate on multi-tool tasks against text and JSON action formats, with about two fewer turns on average.

The mechanism is worth deriving rather than memorising. Suppose a task requires applying the same two-tool sequence to \(n\) items. With one tool call per action, the number of model round trips is \(\Theta(n)\), each carrying the full context (which, by §3, is growing). With code as the action, control flow lives inside a single action:

for f in repo.glob("**/*.py"):
    s = f.read_text()
    if "parse_cfg" in s:
        f.write_text(s.replace("parse_cfg", "load_cfg"))

and the round-trip count is \(\Theta(1)\). Every intermediate result stays in a variable instead of passing through the context window twice. The saving is not a constant factor; it changes the exponent on the thing you are billed for.

Anthropic pushed the same idea into tool discovery. Instead of loading every MCP tool definition into context up front, expose the servers as a filesystem the agent can explore, and let it read only the tool modules it needs, calling them from code. Their reported example goes from roughly 150,000 tokens to roughly 2,000 — a 98.7% reduction — mostly because intermediate results stop making round trips through the model at all.

The catch, and it is a serious one: an action space expressive enough to contain any loop is expressive enough to contain any exfiltration. The decision to hand the model an interpreter and the decision about what that interpreter can reach are the same decision, made once. Teams that make them separately, in that order, months apart, are the ones who end up writing incident reports.

Editing is a protocol, not a talent

The model has decided that line 53 should read bar() instead of foo(). How does that intention become a mutation on disk?

This sounds like a solved problem and is not. There are at least five answers in production use, and they have genuinely different cost and failure profiles:

• Whole-file rewrite. Emit the complete new file. Conceptually bulletproof, and \(O(|F|)\) output tokens for an \(O(1)\) change.
• Search/replace blocks. Emit the old snippet and the new snippet. Cheap, and fails when the search text does not match byte-for-byte, or matches in three places.
• Unified diff. Emit a patch. Compact, and unforgiving about hunk headers, line counts and whitespace.
• Anchored patch formats. Diff-like, but with context anchors instead of line numbers, so the patch survives the model's uncertainty about exactly which line it is on.
• AST transformation. Emit a structural edit. Robust, and requires a parser per language.

Aider is the most useful public case study because it supports several formats and reports how models behave under each. Its headline result is a good demonstration that the protocol is doing real work: switching GPT‑4 Turbo from search/replace blocks to unified diffs took a "lazy coding" benchmark from 20% to 61% — the same model, the same task, tripling its willingness to actually write the code instead of leaving # ...add logic here... behind. The older June GPT‑4 went from 26% to 59% on the same swap. Aider's own explanation is the interesting part: with a diff format, the model behaves more like it is writing structured data for a program to consume than like it is talking to a person. The format changed the register.

It does not always go that way. More recent controlled work isolating edit representation finds that search/replace tends to win for stronger models with unified-diff variants close behind, while smaller open models struggle with all of them; and multi-turn agent evaluations have found whole-file rewriting more stable overall, because a diff that fails to apply burns a repair cycle. There is no universally correct format, which is exactly why real harnesses ship several and pick per model.

The arithmetic of a failed patch

Here is the model that makes the trade-off legible. Let an attempt in format \(F\) cost \(m_F\) output tokens and apply cleanly with probability \(p_F\). Retries are geometric, so the expected output cost of one landed edit is

\[ \mathbb{E}[m_F] \;=\; \frac{m_F}{p_F}, \qquad \mathbb{E}[\text{retries}] \;=\; \frac{1}{p_F} - 1 \]

and the two formats break even when \(m_A/p_A = m_B/p_B\). For a whole-file rewrite, \(m \propto |F|\) and \(p\) is high; for a diff, \(m \propto |{\Delta}|\) and \(p\) is lower and model-dependent. So the break-even file size scales like

\[ |F|^{*} \;\approx\; |\Delta| \cdot \frac{p_{\text{whole}}}{p_{\text{diff}}} \cdot \kappa \]

with \(\kappa\) a small format-overhead constant. Small file, unreliable diff parser: rewrite it. Large file, fluent model: patch it. This is why "which edit format should I use" has no answer that is not a function of three variables.

And note the second-order effect the formula hides. A failed patch does not just cost a retry — it costs a retry plus the tokens of the failure message, appended permanently to a context that is already growing quadratically. The real penalty is worse than \(1/p\).

Figure 7. Interactive. Expected tokens per landed edit under four protocols. The apply-success probabilities are schematic parameters chosen to reproduce the direction of published ablations; the geometry of the trade-off is the transferable part.

Observation design

Now the return path — the \(\varepsilon\) from §2 that nobody writes down.

The agent runs grep -rn TODO . and the environment produces twenty megabytes. What goes into the context? Every option is bad in a different way, and the harness must pick one:

• Everything. Preserves the answer, destroys the context window, and in a long turn destroys the budget too.
• First \(N\) bytes. Cheap, deterministic, and silently discards the answer whenever the answer is late in the output.
• Head and tail. Correct for stack traces, useless for grep. The failure mode moved rather than disappearing.
• Structured summary. "17 matches in 4 files," plus the file list. Costs the harness a parser per tool; saves the model a context window.
• Spill to a handle. Return a summary plus a reference the agent can query again. Best of both, and it requires state.

SWE-agent found empirically that concise search output beat dumping substantial matching context — a finding that reads as obvious and was not obvious at the time. OpenAI's GPT‑5.6 harness post gives the industrial version of the same lesson: tool output is capped at 10,000 tokens by default unless the model explicitly asks for a different limit, precisely so that one tool or one MCP integration cannot unexpectedly consume the context window.

Figure 8. One output, five policies. The green ticks mark the policies under which the single line that explains the bug survives into the model's context.

The right frame is not "the harness pipes output to the model." It is:

\[ o_{\text{model}} \;=\; \Phi\big(o_{\text{env}}\big) \]

with \(\Phi\) a lossy transform chosen by the harness author. This is a sensor pipeline. \(\Phi\) has a gain and a clipping point, and both are decisions. Hand a pilot the raw voltage from a pitot tube and you have not been generous, you have been useless; the instrument's job is to throw away almost everything and keep the one number that matters. An agent's observation encoder has exactly that job, and it is much harder here, because what matters depends on the task and the harness does not know the task.

The practical consequence is that truncation policy is a correctness feature, not a cost feature. A head-truncated pytest output that cuts off before the assertion error has not saved you tokens — it has produced an agent that will now spend six turns guessing.

Verification as a first-class service

A coding agent that stops at "I edited the file" is not an agent, it is an autocomplete with side effects. What makes the loop worth having is that some of the arrows go through the world:

\[ a_t \;\longrightarrow\; o_{t+1} \;\longrightarrow\; a_{t+1} \]

That middle arrow is the only one in the entire system that carries information the model did not already have. Everything else — reasoning, re-reading its own patch, restating the plan — is a rearrangement of what was already in the context. The environment is the sole source of new bits.

Figure 9. Confidence and correctness are different variables. Without an executable check they are also uncorrelated, and only one of them is observable to the agent.

This argues for something stronger than "give the agent a shell so it can run tests." It argues that verification should be a service the harness provides, with a normalised return type:

run_tests(scope)  →  { passed: 418, failed: 2, failures: [...] }
run_linter()      →  { errors: [...], warnings: [...] }
typecheck()       →  { diagnostics: [...] }
build()           →  { ok: false, first_error: {...} }

rather than eight thousand lines of console output that the model has to parse under a truncation policy that was designed for grep. The normalisation is where the value is: it makes the verdict cheap to read, cheap to compare across turns, and — this matters later — cheap for the harness itself to interpret when deciding whether the turn is actually finished.

There is a broader principle here about what makes coding an unusually good domain for agents. It is not that code is easy. It is that code comes with a cheap, automatic, high-fidelity oracle. Most agent domains do not have one, which is why "run the tests" is a luxury and why so much agent research is really environment engineering in disguise.

Making invalid states unreachable

SWE-agent's editor runs a syntax check on the result and refuses edits that break the file. That single behaviour encodes a principle that generalises to the entire harness:

This is the agent-level analogue of constrained decoding. In constrained decoding you mask the logits so ungrammatical tokens cannot be sampled; the model is not asked to remember the JSON schema, it is prevented from violating it. At the harness level the same move is available at two points. Before execution:

\[ g_{\text{pre}}: \mathcal{A} \to \{\text{allow}, \text{deny}, \text{ask}\} \]

validating arguments against a schema, parsing the diff before applying it, compiling the result before writing it, rejecting a command that matches a destructive pattern. And after execution:

\[ g_{\text{post}}: \mathcal{O} \to \mathcal{O} \]

redacting secrets out of tool output before it enters the context, scanning fetched web pages and network-enabled tool results for injected instructions, capping the size of what comes back.

The post-execution guard is the one people skip, and it is the one that has been catching real attacks. A trusted tool is not the same as trusted data: an audited GitHub connector can load a poisoned README straight into the model's context without failing a single malware check. Anthropic's engineering write-up is blunt about the asymmetry — they err toward live inspection of network-enabled tool returns, because once a poisoned result has steered the agent into exfiltrating something, the log shows a successful, authorised API call and there is no after-the-fact signal to find. The classifier doing that inspection can be small and fast; it does not need to be the model doing the reasoning.

The general shape is worth naming, because it recurs: probabilistic defences shape what the agent tends to do; deterministic defences bound what it can do. You need both, and only one of them holds when the other is having a bad day.

Containment and blast radius

Start from the definition of risk:

\[ \text{Risk} \;=\; \underbrace{P(\text{failure})}_{\text{likelihood}} \;\times\; \underbrace{D(\text{failure})}_{\text{damage}} \]

Better models, better evaluations and better classifiers all push on the first term. Nothing about model quality touches the second. And the second term grows monotonically with access: the more an agent can reach, the more a single bad decision costs. So as capability and access both increase, \(P\) shrinks and \(D\) grows, and the product does not automatically go anywhere good.

Anthropic's engineering account of containing Claude across their products puts the strategic conclusion first: rather than supervising what the agent does, supervise what it is able to do. That is a different engineering problem, and a much more tractable one, because it can be solved with primitives that predate agents entirely — process sandboxes, virtual machines, filesystem boundaries, egress control.

Figure 10. Three containment patterns and two ways they still failed. The capability ceiling bars are qualitative.

Three patterns, chosen by the user rather than the model

The three shipped patterns are instructive because the differences between them are not about model capability at all.

The ephemeral container runs code server-side in a gVisor sandbox with a per-session filesystem and no access to the user's machine. Minimal blast radius, and a correspondingly low ceiling on what the agent can do — there is no persistent workspace to build anything in.

The human-in-the-loop sandbox runs on the developer's machine with an OS-level boundary (Seatbelt on macOS, bubblewrap on Linux): reads allowed, writes confined to the workspace, network denied by default. This is only tractable, as the write-up notes, because the typical user is a developer who can read bash and already runs npm install from untrusted sources every week.

The sealed VM gives the agent its own kernel, filesystem and process table via the platform hypervisor, mounts only the workspace the user selected, and keeps credentials in the host keychain so they never enter the guest. This is the pattern for users who cannot be asked to evaluate find . -name "*.tmp" -exec rm {} \; — which is most users.

The selection rule that falls out is not technical:

Two failures worth memorising

The sandbox posts that are worth reading are the ones that describe what broke. Two are directly relevant to harness design.

Exfiltration through an approved domain. A malicious file in a mounted workspace carried hidden instructions plus an attacker-controlled API key. The agent followed them, read other files, and uploaded them using Anthropic's own Files API. The egress proxy checked the destination, saw api.anthropic.com — a domain the product cannot function without — and let it through. The sandbox worked perfectly and the data left anyway.

The reframing is the valuable part: an allowlist is not a destination filter, it is a capability grant. Every function reachable through any allowed domain is now inside your trust boundary. Allowing the API meant allowing file uploads to arbitrary accounts. The fix was a man-in-the-middle proxy inside the VM that only passes requests carrying the VM's own provisioned session token — placed inside because only the inside knows provenance; from the server's view a request from the sandbox looks like any other API client.

Everything before the trust dialog. A developer clones a repository to review a pull request. The repository contains a .claude/settings.json defining a hook. Claude Code read project settings during startup — before showing the "do you trust this folder?" prompt — so the attacker's hook executed automatically. Three separate reported vulnerabilities had this same shape: input from a not-yet-trusted directory parsed before the trust boundary existed.

The generalisable rule: treat project-open, config-load and localhost listeners the way you would treat any inbound request from the internet. They are not trustworthy just because they feel local and arrive early.

And the meta-lesson across all of it, which is the oldest lesson in security wearing new clothes: the hypervisors, syscall filters and container runtimes held. The custom allowlist proxy did not. The software you wrote yourself is the weakest layer, because it has survived the least adversarial attention.

Permissions and the arithmetic of fatigue

Sandboxing answers "what can happen." Permissions answer a different question: "what is the agent allowed to do without asking?"

The natural taxonomy is by reversibility rather than by tool:

• Read-only. ls, grep, git status, reading a file. Free.
• Workspace mutations. Editing a file, running tests, creating a branch. Recoverable, usually via git.
• External side effects. git push, sending mail, deploying, deleting a cloud resource. Not recoverable by anything the harness controls.

The harness's job is to compute \(P(\text{require approval} \mid a_t, s_t)\) and to do it in an order that cannot be subverted. The Claude Agent SDK documents a six-step evaluation: hooks run first and can deny outright; then deny rules, which block even under a bypass mode; then ask rules, which route to the approval callback even under a bypass mode; then the permission mode; then allow rules; and finally, if nothing has decided, a callback to a human.

Figure 11. Interactive. Pick an action and a mode; watch the ladder resolve. The ordering is the security property: no permissive mode can silently override a rule you wrote down.

Notice what that ordering buys. If mode were checked before deny rules, then a single --dangerously-skip-permissions would erase every policy the team had agreed on. Putting deny and ask above mode means the escape hatch is bounded by the policy rather than the other way round. That is a small design decision with a large blast-radius consequence, and it lives entirely in the harness.

Why more prompts means less supervision

Here is the finding that should reorganise how you think about approval flows. Anthropic's telemetry showed users approved roughly 93% of Claude Code permission prompts. And they observed the obvious corollary: the more approvals a user sees, the less attention they pay to each one.

Model it. Let \(a(n)\) be the probability that the reviewer actually reads and correctly evaluates the \(n\)-th prompt of a session, and assume it decays:

\[ a(n) \;=\; a_0\,2^{-n/h} \]

with \(h\) a vigilance half-life measured in prompts. If exactly one action in the session is dangerous and it is equally likely to be any of the \(N\) prompts, the expected probability of catching it is the average

\[ \bar{a}(N) \;=\; \frac{1}{N}\sum_{n=1}^{N} a_0 2^{-n/h} \;=\; \frac{a_0}{N}\cdot\frac{2^{-1/h}\big(1 - 2^{-N/h}\big)}{1 - 2^{-1/h}} \;\xrightarrow[N \to \infty]{}\; 0 \]

which is a slightly formal way of saying something everyone already knows: an approval dialog you see two hundred times a day is not a control. Worse, \(\bar a\) is decreasing in \(N\), so a harness that asks about more things provides less oversight per dangerous action, not more.

Figure 12. Interactive. Vigilance decay, and what removing prompts does to it. The 93% approval rate and the 84% prompt reduction are reported figures; the decay curve's shape is a model, not a measurement.

This inverts the intuitive design. The right move is not to ask about more things, it is to tighten the perimeter so you can ask about fewer. Shipping the OS-level sandbox reduced permission prompts by about 84%, and the prompts that remain land on a reviewer who is not yet numb. Anthropic's reference devcontainer exists for exactly this reason: a tight enough perimeter means the agent can run unattended, with no per-action approvals at all.

Two honest caveats. First, the model-based auto-approval layer that replaced some of those prompts is itself probabilistic — Anthropic reports it blocks roughly 0.4% of benign commands while roughly 17% of overeager actions still get through, which makes it a layer of defence-in-depth inside a sandbox, not a substitute for one. Second, and more uncomfortable: none of this helps when the malicious instruction arrives through the user. In a controlled internal exercise, a researcher phished an employee into launching Claude Code with a prompt that quietly asked it to read ~/.aws/credentials and POST them out; across 25 retries the exfiltration completed 24 times. Model-layer defences anchor on user intent, and when the user is the one typing the instruction there is nothing anomalous to detect. Only egress control and filesystem boundaries held.

Budgets, stopping, and the illusion of done

The loop has no natural length. Suppose at each step the model requests another tool call with probability \(q\). Then turn length is geometric:

\[ P(L = n) = (1-q)q^{\,n-1}, \qquad \mathbb{E}[L] = \frac{1}{1-q}, \qquad P(L > B) = q^{B} \]

The mean is a pole at \(q = 1\). Between \(q = 0.90\) and \(q = 0.98\), \(\mathbb{E}[L]\) goes from 10 to 50 — a fivefold cost increase from a behavioural drift that no evaluation would flag. This is why a turn cap is a harness primitive and not a request you make of the model in the system prompt. Claude Code exposes a max-turns bound; the Agent SDK exposes a dollar budget. The general set is \(B_{\text{turns}}, B_{\text{tokens}}, B_{\text{tool calls}}, B_{\text{wall clock}}, B_{\$}\), and a serious harness enforces several.

Figure 13. Interactive. Turn length as a geometric variable, and what a cap does to its tail.

But a cap is not a stopping criterion. It bounds the bill; it says nothing about whether the work is done. And the default stopping criterion — the model emitted an assistant message with no tool call — certifies only that the model stopped asking for tools.

A better termination predicate for a coding agent is a conjunction:

\[ \text{done} \;=\; \big(\text{no tool call requested}\big) \;\wedge\; \big(\text{verifier green}\big) \;\wedge\; \big(\text{no unresolved plan item}\big) \]

and the interesting thing about that formula is that only the first conjunct comes from the model. The other two come from harness-owned state: a test runner, and an explicit plan. This is one of the strongest arguments for making the plan a first-class harness object rather than free text in the model's reasoning — Codex, for instance, ships an update_plan tool alongside its shell tool. A plan the harness can read is a plan the harness can check against before letting the turn end.

How much does this matter in practice? §18 has the numbers, and they are worse than you would guess: in one controlled study, one harness declared itself finished on a failing test suite in 14 of 100 trials, while another harness running the same models never did so once.

The event log is the state

Naive harnesses keep an agent_state object and mutate it. Serious ones invert that: record an append-only stream of typed events and derive state from the stream.

UserMessage(text)          ModelResponse(items)
ToolCall(name, args)       ToolResult(call_id, output, exit_code)
FileEdited(path, diff)     TestsRun(passed, failed)
ApprovalRequested(action)  ApprovalGranted(action, scope)
PlanUpdated(items)         Compacted(from, to, summary_id)

OpenHands built its architecture around exactly this: an event stream as the chronological action/observation history, with the V1 SDK making it an event-sourced state model with deterministic replay. Google's Agent Executor generalises the idea into infrastructure — durable execution for agents, harnesses, tools, skills and sandboxes via an event log and snapshotting, with connection recovery that backfills from the last sequence a client saw, a single-writer architecture to keep distributed session state consistent, and trajectory branching so you can explore two decision paths from the same checkpoint.

The payoff list is long and mostly unglamorous: replay, debugging, persistence, auditing, resumability across a client restart, and the ability to answer "why did it delete that file?" three days later. But two entries on that list deserve emphasis.

Branching. Once state is derived from a log, "what if it had chosen differently at step 12?" becomes a supported operation rather than a rerun from scratch. That is the difference between debugging an agent and re-rolling the dice at it.

Trajectories. An event log is a training trajectory. The same structure that lets you audit a production run lets you turn that run into RL data. Which is the bridge to §21, and the reason the harness is not only a deployment concern.

Git deserves a mention here as the luxury coding agents have and most agents do not. A version-controlled workspace makes a large class of actions partially reversible: the transition \(s_t \xrightarrow{a_t} s_{t+1}\) can often be undone by git checkout. Reversibility changes the calculus of exploration. An agent that can cheaply revert is allowed to try things; an agent operating on a production database is not. A harness that checkpoints before risky edits is buying the model permission to be wrong, which is worth more than most prompt engineering.

Compaction: the lossy codec in the loop

The context is monotone (§2) and the window is finite. Eventually \(|C_k| > C_{\max}\), and something has to give. The options are a compression problem in disguise:

• Drop old turns. Cheap, and it is exactly the early turns that contain the task definition.
• Summarise. Lossy in an uncontrolled way — you find out what the summariser discarded when the agent asks a question it already answered.
• Retrieve. Keep everything externally, pull back what is relevant. Now you have a retrieval problem over your own history.
• Compact. Replace the input with a smaller representative list and continue.

Codex's evolution here is a nice miniature of the whole field. It began with a manual /compact command that queried the model with the conversation plus summarisation instructions and used the resulting assistant message as the new input. It now uses a dedicated compaction endpoint automatically once a configured token threshold is exceeded, returning a list of items — including an opaque encrypted item intended to preserve the model's latent understanding of the original conversation — that replaces the previous input.

That last detail is quietly radical. It concedes that a natural-language summary is a lossy codec for a conversation, and tries to carry something across the boundary that is not expressible in the summary. Whether that works is an empirical question I have not seen cleanly answered in public.

Compaction is where this article hands off to the next one in the series. The write problem — what earns a place in the compacted state, what should be forgotten, what should be updated rather than appended — is the subject of Memory Is More Than Context. For now, note only the harness-design consequence: compaction is a scheduled cache-miss. It rewrites the prefix by construction, so the turn immediately after a compaction pays full price for its entire context. If your threshold is wrong you can spend more on repeated compactions than you saved.

Progressive disclosure and the tool-context tax

Every tool definition is text in the prompt, on every call of the turn. With a handful of tools this is invisible. With a dozen MCP servers exposing fifty tools each, it is the dominant cost before the agent has done anything at all.

Write the tax down. With \(M\) tools averaging \(\tau\) tokens of schema, and a turn of \(K\) steps, the schema cost alone is

\[ T_{\text{tools}} \;=\; K \cdot M \tau \]

linear in the number of tools and multiplied by the loop length. At \(M = 300\), \(\tau = 120\) and \(K = 40\) that is 1.4 million tokens spent describing capabilities, most of which will not be used.

Two structurally different fixes are in production. The first is deferred discovery: OpenAI's harness makes integrations, custom MCP tools, skills and plugins surfaceable only when needed, rather than loading them all up front. The second is filesystem-shaped tools: expose servers as directories the agent can list and read, so it loads the definitions for the two tools this task needs and never sees the other 298 — the pattern behind the 150k-to-2k figure from §7.

Skills are the same idea applied to procedures rather than APIs. Codex puts skill metadata — names and descriptions — into the initial prompt alongside project instructions, with the body fetched only when the model decides to use one. Which sets up a spectrum worth thinking about explicitly when you design a harness:

The design question for any given behaviour is: how far right can I push it? "Always run the linter after editing" can live at any point on that line. As a prompt it is advice the model may forget under context pressure. As a skill it is a procedure loaded when relevant. As a tool it is an action with a schema. As code in a post-edit hook it is a fact about the universe. Each step right costs engineering and buys determinism, and the harness author's judgement is mostly about where on that line each behaviour belongs.

The harness effect, measured

Everything so far has been mechanism. Now the empirical question: hold \(\pi_\theta\) fixed, vary \(H\), and see what moves.

A controlled 2026 study did exactly this — two recent agentic coding models, three open-source harnesses (Goose, OpenCode, OpenHands-SDK), fifty stratified tasks from Terminal-Bench Pro, 300 trials, identical task manifests, a largely identical system-prompt template, and the same sandbox and wall-clock cap. The only things left free were each harness's native tool API, its automatic context pre-loading, and its internal retry and sub-agent logic. In other words: the harness.

The results split cleanly along two axes, and the split is the finding.

Pass rate barely moved. Within a model, the harness range is 2 to 8 points, and most pairwise differences sit inside a 95% paired-task bootstrap interval that includes zero. If you only report pass rate, the harnesses look interchangeable.

Cost moved by 40×. Tokens per solved task ran from 28k to 1.55M. And crucially, this was not a turn-count effect: average turns differed by about 1.2×. The gap comes from per-turn context volume — from what each harness pre-loads, how it serialises tools, and how many turns it spends doing nothing. On the same data, upgrading the model changed tokens per solved task by 1.0 to 1.3×.

Sit with the asymmetry. Harness choice shifted cost by 40× and accuracy by at most 8 points. Model choice shifted accuracy by 4 to 10 points and cost by almost nothing. They are levers on different quantities. Anyone optimising a deployment by picking the model first and the harness second is tuning the wrong knob.

The same asymmetry shows up on the public benchmark. The Terminal-Bench 2.0 paper reports one model at 52.1% under one harness and 57.8% under another — a 5.7-point gain — while consuming 256.9M versus 3.9M input tokens, a 65× difference. Which is why Terminal-Bench ranks harness–model pairs rather than models.

Figure 14. Interactive. The same six pairs on a cost–accuracy plane. Drag your token budget across two orders of magnitude and watch the "best" system change identity.

Failure fingerprints

The second finding is stranger and more useful. Each harness fails in a characteristic way, and the pattern replicates across both models — which identifies it as a property of the scaffold rather than the weights.

Goose is dominated by voluntary stops: it gives up rather than commit to a wrong solution, and never once declared a task done on a failing suite. OpenHands-SDK pushes toward closure and, in 14 of 100 trials, called finish while the verifier disagreed — plus another 12 that ran out of turns. OpenCode never declared false victory either, but with no turn budget exposed it burned the wall clock instead.

Figure 15. Outcome distributions, 100 trials per harness pooled over two models. Same solve rates, completely different failure shapes.

This is a richer signal than pass rate because it prescribes different human behaviour. If your harness stops early when stuck, you should budget for finishing tasks yourself and can trust its positive verdicts. If your harness drives to closure, you must verify every output, because "done" is a claim rather than a measurement. Those are opposite review disciplines, and nothing on a leaderboard tells you which one you signed up for.

It also connects back to §14. The harness with an exposed turn budget converts what would have been a wall-clock hang into a clean MAX_TURNS signal — the failure is legible instead of just slow. Legibility of failure is a harness feature.

Anatomy across real harnesses

Pulling the mechanisms together against real systems. The point is not to rank them — they were built for different jobs, on different bets about what the model still needs — but to show that every one of the axes in this article is a place where real systems make different choices. Ten systems: four vendor products, three lab-shipped runtimes, a minimalist harness, and two research agents.

^† dsh reads both CLAUDE.md and AGENTS.md, which in repositories that keep the two files identical for cross-tool compatibility injects the instruction set twice. Compiled from public documentation, papers and hands-on reports as of September 2026; several of these projects ship weekly, so treat any single cell as a pointer to go and check rather than as a citation.

Four observations from the grid.

First, nobody does everything, and the gaps are not oversights. Aider is the only system with several edit protocols because editing is its central concern. Pi ships four tools and no approval gate because its authors argue that approval prompts produce fatigue and end up as security theatre — a position the fatigue model in §13 makes quantitative rather than rhetorical. mini-SWE-agent deliberately has almost nothing. Each blank cell is a thesis about what the model no longer needs.

Second, the axes split into two clusters. The rows down to progressive disclosure are about helping the model act well. The rows from the permission layer down are about bounding, recording and paying for what it does. Products score high on the second cluster; research artifacts and minimalist harnesses score high on the first. That split is the subject of the next section.

Third, the loop is never the interesting part. Two systems here implement the core loop in about a hundred lines, and one of them is competitive on SWE-bench Verified. At the other extreme, DeepSeek's harness makes the loop a mounted plugin sitting at the same privilege level as the tool registry — core/agent holds the interface and core/agent-loop holds the default driver, so replacing the driver is a configuration change rather than a fork. Whether the loop is nine lines you can read or a package you can swap, it is never where the difficulty lives. Everything else is the four functions it calls.

Fourth, and this is new since 2025: every serious model lab now ships its own harness. Alibaba's Qwen Code and ByteDance's Trae Agent arrived in mid-2025, Moonshot's Kimi CLI that October, Zhipu's ZCode in July 2026, and DeepSeek's dsh in August 2026, alongside xAI open-sourcing Grok Build. That convergence matters for a reason worth stating plainly: a lab that publishes agentic benchmark numbers without publishing the harness has published an unreproducible number. On the argument of §18, that is not a nice-to-have. If \(Q=f(\pi_\theta,H,\mathcal{A},\mathcal{E})\) and a forty-fold token spread hides behind a single leaderboard row, then releasing \(\pi_\theta\) without \(H\) is releasing one argument of a four-argument function.

Two labs land on opposite sides of that. DeepSeek's harness release was, by its own framing, partly a response to criticism of vendor-reported, unreproducible agentic scores — its official agent numbers were produced in the harness's minimal mode, which a third party can now rerun. Right direction, gap not yet closed: V4-Pro's SWE-bench Verified score circulates as 80.6% self-reported against 96.4% from a third-party evaluator, with no authoritative reconciliation. Nobody has reproduced and disproven an official number; the objection is about transparency, and an open harness is exactly what makes the objection answerable.

Qwen Code shows what the answered version looks like. Its repository publishes benchmark runs as versioned release artifacts: dataset and revision, harness version down to the commit, model, the dispatch workflow that ran it, resolved and unresolved counts, execution errors and infrastructure failures separated out, an explicit score denominator, and a tarball of per-case trajectories. One such release reports 76.16% on SWE-bench Verified — 377 resolved of 495 valid grader results, with two execution errors and three infrastructure failures excluded and named — from Qwen Code v0.21.15 driving qwen3.7-plus. Runs that fail their own quality gate are published too, marked quarantined and unscored. Whatever one makes of the number, that is a harness–model pair with its trajectories attached rather than a leaderboard row, and it is close to the reporting format the rest of §18 argues the field needs.

Two harnesses in the grid are worth reading against each other, because they are the extreme points of the same axis. Pi gives the model four tools — read, write, edit, bash — and the shortest system prompt of any agent in wide use, on the bet that a frontier model has already been trained on this job and further instruction is noise. DeepSeek's dsh makes every component swappable at runtime, including compaction, the sandbox and the loop. One hands-on comparison put uncached input at roughly 47.6k tokens for dsh against about 4.5k for Pi on the same model. That is an order of magnitude of prompt paid for configurability, and it is the same trade the cost model in §3 describes: a large static prefix is cheap per call once it caches, and ruinous across a long turn if anything upstream of it moves.

Two design decisions in dsh are worth lifting out because they are direct implementations of arguments made earlier in this article. Its session log is append-only under a runtime-enforced invariant — anything the model can see must be reconstructible from the log — with the visible history projected from the log rather than stored separately, so resume, fork, search and replay fall out of the representation instead of being retrofitted; that is §5 and §15 as an architectural guarantee rather than a convention. And compaction is itself a plugin that brackets its work in three log events, so a crash mid-compaction leaves a reconstructible state — which is what taking §16's "compaction is a scheduled cache-miss" seriously actually looks like in code.

The security layering in dsh is also instructive in the direction of §12. Tool execution is sandboxed properly — bubblewrap plus Landlock behind a fail-closed launcher on Linux, Seatbelt on macOS, restricted ACL tokens on Windows, with the approval model a closed enumeration so anything unrecognised is refused rather than allowed. But plugins run inside the harness process, and the plugin registry passed two thousand submissions within two days of launch. The boundary is real and it is drawn in the wrong place for the threat that actually materialised: a third-party plugin has the harness's own filesystem and shell access, which makes it closer to an npm postinstall script than to an MCP server. This is the same shape as the allowlist failure in §12 — the primitive held, and the thing bolted beside it did not.

The moving frontier

Now the complication that keeps this from being a tidy story.

In 2024, SWE-agent's whole thesis was that agents need purpose-built interfaces, and the numbers backed it: 12.5% versus 3.8%, with 10.7 points attributable to the interface. In 2025, the same Princeton and Stanford group shipped mini-SWE-agent: about a hundred lines of Python, no tools other than bash, not even using the tool-calling interface of the models, a completely linear message history, every action run through an independent subprocess.run with no persistent shell session. It scores over 74% on SWE-bench Verified.

Their own framing is refreshingly direct: back then they placed a lot of emphasis on tools and special interfaces; a year later, as models became more capable, a lot of that turned out not to be needed to build a useful agent.

So which is it? Both, and the resolution is that "the harness" is not one thing.

2026 sharpened the question rather than settling it, because the field ran the experiment at both extremes at once. Pi pushed the mini-SWE-agent thesis into a product: four tools, the shortest system prompt of any agent in wide use, no approval gate, on the explicit argument that a frontier model has already been trained on this job and that prompting it further is noise. In the same year, DeepSeek's dsh pushed the opposite way, making every component — adapter, tools, sandbox, compaction, UI, and the loop itself — a runtime-mountable plugin. Roughly ten times the uncached input, for configurability rather than for capability.

What is striking is that these are not disagreements about the same axis. Pi is minimal on the interface half and explicitly opts out of the governance half. dsh is maximal on the governance half — enforced append-only logs, fail-closed sandboxing, a closed approval enumeration — and does not claim any interface advantage at all; its own documentation points elsewhere for daily coding. Neither is a counterexample to the split. They are the split, drawn as two products.

Figure 16. Schematic. Which responsibilities the harness must supply, over time. Trend directions follow the published record; the cell values are an illustration.

Split the harness into two halves and the apparent contradiction dissolves.

The competence half — file viewers, search shaping, edit scaffolding, retry prompting, decomposition hints — exists to compensate for things the model cannot yet do. It is, by construction, temporary. Every capability improvement erodes it. Anything in this half that you build today is depreciating.

The governance half — sandboxing, permissions, egress control, budgets, event logs, tracing, verification, resumability, compaction — does not compensate for a model weakness. It exists because the agent is a program running on a real machine with real credentials and a real bill. A more capable model does not need less of this. It needs more, for a reason Anthropic states directly: less capable models misread situations and make obvious errors, while more capable models make fewer mistakes but are better at finding unexpected paths to a goal, routing around restrictions nobody thought to write down.

This has a practical corollary for anyone choosing what to build. Custom retrieval heuristics, elaborate prompt scaffolds and hand-tuned tool wrappers have a short half-life. Containment, budgets, event logs and verification services do not. If you have limited engineering time, spend it on the things that keep working when the model gets better.

Harness and model co-evolve

The last twist is that the two halves are not independent, and the causal arrow now runs both ways.

Historically the direction was one-way: build a better harness around a fixed model. Today it is a cycle. The harness generates trajectories; those trajectories become RL data; the model gets better at operating that harness; and then the harness is redesigned around the model's new competence. Formally, you are no longer optimising \(\pi_\theta\) for fixed \(H\), you are doing something closer to

\[ \max_{\theta,\,H} \; \mathbb{E}_{\tau \sim \pi_\theta \circ H}\big[R(\tau)\big] \]

over a joint space where one argument is neural and the other is a codebase. Which produces two consequences worth flagging.

The first is model–harness fit. If a model is trained on trajectories from a particular action space and edit protocol, it will be better in that action space than in an equally good alternative. This is a plausible partial explanation for why the same weights score differently under different scaffolds — it is not only that one scaffold is better engineered, it is that one of them matches the training distribution. It also predicts something uncomfortable: harness diversity has a cost that grows as labs train harder against their own scaffolds.

The second is automated harness search. A 2026 line of work now treats the harness itself as the optimisation target. Meta-Harness gives a coding agent unrestricted filesystem access to the traces, scores and source of prior attempts and lets it rewrite the harness — searching over complete executable procedures rather than a predefined space of context-management strategies, and the authors note the workflow only became practical once coding agents got good enough to do the searching. Related work evolves harnesses from observability data, or trains a controller that sits between model and scaffold. There is even a small literature now on how to evaluate harness evolution without fooling yourself.

Reading that literature carefully turns up a sobering pattern, though: the improvements found by automated scaffold search concentrate in software-engineering hygiene — parsing, retries, dispatch, output extraction — and rarely deliver domain reasoning the base model could not produce given a good enough prompt. Which is consistent with everything else in this article. The harness is not where intelligence comes from. It is where reliability comes from, and reliability is what turns a capable model into a system you can deploy.

What is still unsettled

Six things I do not think anyone has resolved, stated as questions rather than as positions.

Where does the competence half stop shrinking? The evidence that interface scaffolding is being absorbed into weights is strong. Whether it goes to zero is unknown. Repository representation in particular may be a permanent harness responsibility: no context window makes a fifty-million-token monorepo free, and the retrieval signal is structural rather than semantic in a way that does not obviously become a model capability.

Is compaction losing more than we can see? Carrying an opaque encrypted representation across a compaction boundary is an admission that natural-language summaries are a poor codec for a conversation. Nobody has published a clean measurement of what a compaction costs in task success, which is a strange gap given how routine the operation now is.

Do subagents raise or lower trust? A subagent can isolate untrusted content and return structured facts rather than raw text, which is a real security win. But if a parent treats a child's output as higher-trust because it "came from us," you have manufactured a new injection path. The trade-off between differentiated trust levels and trust escalation is real and, as far as I can tell, unsolved.

How much of the harness effect is fit rather than quality? Separating "this scaffold is better engineered" from "this scaffold matches the model's training distribution" requires experiments that the labs can run and the rest of us cannot. The distinction matters enormously for anyone choosing a harness for a model they did not train.

What is the right unit of evaluation? The case for ranking harness–model pairs with cost, idle turns and failure-category vectors alongside pass rate seems overwhelming to me. It is also more expensive to produce and harder to summarise in a headline, which is usually what decides these things.

Whose identity does the agent act under? Should an agent have its own principal identity, or inherit the user's permissions? Anthropic's current answer for one product is concrete — credentials stay in the host keychain, the sandbox gets a scoped, separately revocable session token — but the general cross-platform question is open, and it is the question that decides what "audit" means for a fleet of agents.

If there is one thing to carry out of all this, it is the reframing at the top. A benchmark row that reads "model X: 57.8%" is not measuring a model. It is measuring \(f(\pi_\theta, H, \mathcal{A}, \mathcal{E})\) and reporting only the first argument. Once you have seen the other three, the number stops being a fact about intelligence and starts being a fact about a system — which is, after all, what you are actually going to deploy.

References

Grouped by topic. Every entry links to its primary source.