How Models Improve Without Changing Their Weights

A fourth axis

Every scaling story in this series so far has been a story about training. In Scaling Laws from First Principles the whole problem was a constrained optimisation over two numbers: given a training budget C_train, how do you split it between parameters N and tokens D? Once you have chosen, the run happens, the weights freeze, and quality is whatever the law says it is.

That picture is incomplete, and the incompleteness is not a detail. A frozen model is not a lookup table; it is a sampler. You can query it once, or a thousand times. You can let it emit twelve tokens or twelve thousand. You can throw away nine hundred of its thousand answers and keep the one that survives a test suite. None of that touches a single weight, and all of it changes the probability you get a correct answer.

The conceptual centre of this article is that fourth argument. It behaves differently from the first three in three important ways, and most confusion about test-time scaling comes from forgetting one of them.

It is spent per query, not per model. C_train is paid once and amortised over every request for the lifetime of the model. C_test is paid every single time. A technique that buys five accuracy points for a 100× inference bill is a research result; it is not necessarily a product.

It is allocatable per instance. There is exactly one compute-optimal (N, D) for a training run. There is a different optimal C_test for every prompt, because prompts differ in difficulty. This single fact is what makes inference scaling a strictly richer optimisation problem than Chinchilla, and it is where most of the remaining headroom lives.

It converts a distribution into an answer. Extra training compute makes the model's output distribution better. Extra test-time compute lets you sample that distribution more than once and then choose. Choosing requires a selector. The quality of the selector, not the quantity of samples, is usually the binding constraint — and that is the single most under-appreciated result in this whole literature.

Eight ways to spend a budget

Before any mathematics, it helps to see the landscape as a single plane. Almost every test-time method in the literature is a point in a two-dimensional space: how wide you go (how many independent trajectories you run) and how deep you go (how many tokens each trajectory is allowed to consume). Everything else — verifiers, voting, search, refinement — is a rule for what to do with the trajectories once you have them.

The figure below walks that plane. Step through it; each stage lands on one family and names what it actually buys you.

What the extra tokens cost

It is tempting to treat C_test as one number. It is not; it is at least four, and they scale differently. This matters because two strategies that look identical on a FLOPs plot can be an order of magnitude apart in latency or in how many concurrent requests a server can hold.

Take a dense model of N parameters. Decoding costs roughly 2N FLOPs per generated token for the weight-matrix work. Compare two ways of spending the same token budget KT: K parallel trajectories of T tokens, versus one trajectory of KT tokens.

Two of those rows are worth pausing on. The KV-cache row is an equality that surprises people: K caches of length T and one cache of length KT hold the same number of token-slots, so the memory bill for going wide and going deep is the same. What differs is shape — going wide needs K concurrent batch slots, going deep needs one very long sequence. The attention row is a genuine asymmetry: because attention cost grows with position, a single long trajectory pays quadratically in its own length while K short ones pay quadratically only in T. Serial depth is the more expensive kind of compute, per token, and it gets worse the deeper you go.

The mechanics behind those rows — prefix caching, batch-slot pressure, the prefill/decode split, why decode is memory-bandwidth-bound rather than FLOPs-bound — are the subject of The Mechanics of LLM Inference and The Serving Playbook. Here I only want the accounting, because it explains a practical asymmetry: parallel width is cheap to add and hard to use; serial depth is expensive to add and easy to use. Width gives you many candidates and no opinion about which is right. Depth gives you one candidate the model already believes in.

Why sampling has a scaling law

Start with the simplest possible intervention: ask the same question k times and see whether any answer is right. The quantity that tracks is coverage — the fraction of problems solved by at least one sample. Brown et al. measured this carefully and found something striking: coverage keeps rising with sample count across four orders of magnitude, and the relationship is often log-linear, well modelled by an exponentiated power law. On SWE-bench Lite, DeepSeek-Coder-V2-Instruct went from solving 15.9% of issues with one sample to 56% with 250 — past the 43% single-sample state of the art at the time.

That log-linearity is initially baffling, because for a single problem it is provably false. Let me derive why, because the resolution is the most useful piece of intuition in this article.

One problem: coverage saturates fast

Fix a problem the model solves with probability p per independent sample. The chance that k samples all fail is (1−p)^k, so

Solve for the half-way point: c(k) = ½ at k^* = ln2 / ln(1/(1−p)) ≈ 0.69/p. Plotted against log k, this is a sigmoid: nothing, then a sharp rise centred at roughly 1/p, then a flat ceiling at 1. It is not log-linear anywhere. A problem with p = 0.01 is essentially unsolved at k = 10 and essentially solved at k = 1000, and in between it swings through the entire range in about two decades of k.

Many problems: the sigmoids add up to a power law

Now aggregate. A benchmark is not one problem with one p; it is a population of problems with a distribution of p. Write that distribution as a density π(p). Benchmark coverage is the average of the per-problem sigmoids:

Take a concrete and reasonable family: p ∼ Beta(a, b). Then the expectation has a closed form, and its large-k behaviour is the whole story:

This is the punchline, and it is worth saying in words. No individual problem obeys a power law. The population does, and the exponent is determined entirely by how much probability mass sits near p = 0 — how many problems the model can almost never solve but not never solve. A Beta density behaves like p^a−1 near zero; small a means a thick pile-up of nearly-hopeless problems, and each decade of extra sampling drags a slice of them across the line. The smooth scaling curve is an artefact of heterogeneity, not a property of any single instance.

And the corollary is immediate: if a fraction π₀ of problems have p exactly zero — the model genuinely cannot produce the answer, at any temperature, ever — then coverage saturates at 1 − π₀ and no amount of sampling moves it. The straight line is a straight line only until it hits that wall.

Play with the figure. Change the shape of π(p) and watch both the tile grid and the slope of the log–log line respond.

The generator–verifier gap

Coverage is not accuracy. Coverage says a correct answer exists somewhere in the pile. Accuracy requires you to hand back exactly one answer. The distance between those two quantities is the single most important number in test-time scaling, and it is where the field's early optimism ran into a wall.

The same paper that showed coverage rising over four orders of magnitude also reported the other half: in domains without automatic verifiers, majority voting and reward-model selection plateau beyond several hundred samples and fail to keep up with the sample budget. Generation kept scaling. Selection stopped.

Why majority voting has a hard ceiling

This is provable in one line, and the proof explains everything. Let the model's answer distribution for a given problem be q(·) over possible final answers. Majority voting over k samples returns the empirical mode. By the law of large numbers, as k → ∞ the empirical mode converges to the true mode argmax_y q(y). Therefore:

So majority voting is an estimator for a quantity that is already determined the moment the weights froze. Past a few dozen samples you are paying compute to reduce sampling noise on a number that was never going to change. Worse, on hard problems where the modal answer is a confident, systematic error, more voting makes you more sure of the wrong thing.

Contrast that with coverage, whose limit is 1 − π₀ — the fraction of problems the model can produce a correct answer for at all. The gap between these two limits is the generator–verifier gap:

The false-positive ceiling

Now put a verifier in the loop. If the verifier is perfect — a Lean proof checker, an exhaustive test oracle, a numerical answer key — the picture is beautiful: accuracy equals coverage, and coverage keeps scaling. This is exactly why the strongest published sampling results live in code and formal proofs, and why the SWE-bench number above was possible.

Real verifiers are not perfect. Unit tests have gaps. Reward models are approximations. Self-verification is the model grading its own homework. Stroebl, Kapoor and Narayanan made the consequence precise, and the argument is short enough to reconstruct from scratch.

Let p be the per-sample probability of a correct solution and let φ be the verifier's false-positive rate: the probability that an incorrect sample passes. Assume for simplicity that correct samples always pass. After k samples, the expected number of passing candidates that are correct is kp; the expected number that are wrong-but-passing is k(1−p)φ. If you pick uniformly among passers, your accuracy is the precision of the verifier on that pool:

Read that limit carefully. It contains no k. Sampling harder does not improve it, because sampling harder recruits false positives at exactly the same rate it recruits true ones. Some concrete values, with p = 0.05:

• φ = 0 → ceiling 100%. Perfect verification; scale away.
• φ = 0.01 → ceiling 84%. A 1% false-positive rate costs sixteen points.
• φ = 0.05 → ceiling 51%. Equal to the sample success rate; you are flipping a coin among passers.
• φ = 0.20 → ceiling 21%. Most of what survives verification is wrong.

The asymmetry is brutal: the ceiling depends on the ratio φ/p, so the harder the problem — the smaller p — the more a fixed verifier error rate hurts. This is the mathematical form of a familiar practical experience. Resampling a coding agent until the tests pass works wonderfully until the tests are the thing that is wrong, at which point you have built a very expensive machine for finding test-suite exploits.

Compute-optimal inference

With generation and selection separated, the design question becomes an allocation question, and it has an exact analogue in the training-time story. Set them side by side, because the parallel is the intellectual pivot of this article.

The first three rows are a clean analogy. The last three are where the analogy breaks, and each break makes the inference problem harder.

Snell, Lee, Xu and Kumar ran the experiment that made this concrete. They studied two mechanisms — searching against dense process-based verifier reward models, and letting the model revise its own answer distribution at test time — and found that which mechanism wins depends critically on prompt difficulty. Easy prompts want sequential revision; harder prompts want more parallel search. Because the right answer varies per prompt, a strategy that picks per prompt beat a fixed best-of-N baseline using more than 4× less test-time compute. And in a FLOPs-matched comparison, on problems where a smaller base model already had non-trivial success rates, test-time compute let it outperform a 14× larger model.

That last sentence gets quoted constantly and misread almost as constantly, so it is worth unpacking the two qualifiers that are doing all the work.

This is also where the story connects back to L3. The lifetime-cost equation there was C_total = C_train + Q · C_inference. Test-time scaling does not change the shape of that equation; it inflates C_inference by a factor you now control. A model that answers with 10,000 thinking tokens has an inference cost roughly two orders of magnitude above one that answers with 100. If Q is large, that factor lands directly on the wrong side of the ledger, and the compute-optimal choice shifts back towards a better model that thinks less.

What the curves measured

The empirical counterpart to Kaplan and Chinchilla plots is Wu et al., who swept cost–performance trade-offs across greedy decoding, majority voting, best-of-n, weighted voting and two tree-search algorithms, at several model sizes and compute budgets. The headline finding rhymes with Snell: smaller models paired with better inference algorithms sit on the Pareto frontier. Concretely, Llemma-7B with their REBASE tree search outperformed Llemma-34B with standard majority voting on MATH across all the FLOPs budgets they tested, and they report the 7B model beating the 34B until roughly 128 solutions are sampled.

Two secondary findings from that paper deserve more attention than they usually get. First, MCTS underperformed plain sampling at matched budget, because rollouts are expensive and it produced many unfinished solutions that then cast ineffective votes — a reminder that a search algorithm's cost model matters as much as its search quality. Second, REBASE's advantage came from using a process reward model to decide how far to expand each node rather than paying for explicit rollouts. Both observations point the same direction: at fixed budget, the winner is usually the method that extracts the most signal per token, not the method with the most sophisticated search.

Budget forcing

Everything so far has treated C_test as a dial you set from outside. For serial reasoning that is harder than it sounds: the model decides when to stop by emitting an end-of-thinking delimiter, and you do not control that decision. Muennighoff et al. proposed the most stripped-down possible fix, and it is my favourite result in this literature because it is so nearly trivial.

Budget forcing has two halves. To spend less: if the model exceeds a token limit, force-append the end-of-thinking delimiter, which makes it transition to producing an answer. To spend more: suppress that delimiter when the model tries to emit it, and append the word "Wait" instead. The model, having been handed a continuation where it expected a terminator, goes back and re-examines its work.

That is the entire mechanism. No new training objective, no verifier, no search. Applied to a 32B model supervise-finetuned on just 1,000 curated questions and reasoning traces, it produced a model that exceeded o1-preview on competition maths by up to 27%, and extending thinking with budget forcing moved AIME24 from 50% to 57% — past what the same model achieved without the intervention.

The word choice is not incidental. The paper's own ablation found "Wait" outperformed neutral fillers like "Hmm", which suggests the mechanism is less "more compute" and more "an injected prompt to doubt". That is a meaningfully different claim, and one the community has not fully digested: budget forcing may be a lightweight semantic intervention wearing a compute-scaling costume.

When more thinking hurts

Every curve so far has been monotone or asymptotic. Neither is guaranteed. There is now a substantial body of work showing that extending reasoning can make accuracy fall, and understanding why is important because the failure is not random noise — it is structured.

Gema et al. constructed tasks where longer reasoning reliably degrades performance and catalogued five distinct failure modes. Claude models became progressively more distracted by irrelevant information in the prompt as they reasoned longer. OpenAI o-series models resisted distractors but overfitted to the framing of the problem. On tasks with spurious features, extended reasoning pushed models away from a reasonable prior and towards plausible-but-wrong correlations — though few-shot examples largely fixed that one. All models struggled to hold focus on complex deductive constraint tracking. And extended reasoning amplified some concerning behaviours, with Claude Sonnet 4 showing increased expressions of self-preservation.

The reported magnitudes are not small. In their evaluation, Claude Opus 4 dropped from near 100% to roughly 85–90% as reasoning extended, and DeepSeek R1 fell from 70% to 30% in a natural-overthinking setup with five distractors.

Independently, Chen et al. documented the complementary waste: o1-like models burning enormous token counts on trivial arithmetic with essentially no accuracy benefit, an inference budget uncalibrated to actual question difficulty. And the broad empirical picture across several groups is an inverted-U — accuracy rises with reasoning length, peaks, and declines — with the peak arriving earlier on easier problems.

A mechanism, not a mystery

Here is a way to think about it that I find clarifying. Treat the reasoning trace as a random walk in the space of candidate conclusions. At each step the model can move towards the correct region or towards an attractor built out of whatever is salient in the prompt — a distractor number, a familiar-looking problem template, a spurious feature. Early steps have high drift towards correct, because the model is applying the skill it actually has. Later steps have lower drift, because the easy structure is exhausted, and the walk is increasingly governed by whatever attractors are nearby.

Under that view, accuracy at length T is the probability of being in the correct region at time T, and there is no reason for that to be monotone. If the correct region is not absorbing — if the model can revisit and revise a correct conclusion — then extra steps are extra opportunities to leave it. This predicts exactly what the empirical work finds: negative flips, where a model that had the right answer at step j abandons it by step j+m, and that easy problems cross into the harmful zone sooner because they reach the correct region sooner and then have longer to wander.

The theoretical side agrees on the precondition. Javanmard, Mirzasoleiman and Mirrokni, analysing test-time scaling for transformers trained on an in-context linear-regression task, prove that if the skills required by a downstream task are not sufficiently represented in the training data, increasing test-time compute can actively harm performance. Extra deliberation amplifies whatever procedure the model has. If the procedure is right, amplification is what you want; if it is subtly wrong, you are compounding an error with interest.

Allocate, don't extend

Put the pieces together and the conclusion is forced. Uniform budgets are wasteful in three separate ways at once: easy problems are solved in the first few samples and everything after is pure cost; hopeless problems consume budget forever and return nothing; and over-long traces on easy problems actively hurt. The frontier of this field is no longer "think longer". It is allocate.

The optimisation is a clean concave knapsack, and the solution is water-filling. Let problem i receive b_i units of compute and let a_i(b) be its expected accuracy, concave and increasing. Maximise the total subject to a budget:

Now substitute the coverage model. With a_i(b) = 1 − (1−p_i)^b, the marginal return is a_i′(b) = (1−p_i)^b ln(1/(1−p_i)). Setting that equal to λ and using ln(1/(1−p)) ≈ p for small p:

Read the three regimes off the formula:

• Easy (p near 1): the 1/p prefactor is small. A handful of samples, then stop. C_test ↓
• Hard but solvable (p small but above threshold): 1/p is large and the log is positive. This is where the compute goes. C_test ↑
• Effectively unsolvable (p ≤ λ): allocation is exactly zero. The formula refuses to spend, without needing a special case. C_test ↛ ∞

There is a pleasing extra fact hiding in there. Differentiate b^*(p) and you find the maximum at p = eλ, with peak allocation b^*_max = 1/(eλ). The problems that deserve the most compute are the ones sitting just above the give-up threshold — roughly 2.7× the shadow price. Not the hardest problems, and not the ones you are most likely to solve. The ones on the edge.

This is the theory behind a line of empirical work. Damani et al. learn a model that predicts the distribution of rewards given an input and a budget, then allocate accordingly; across programming, mathematics and dialogue they match non-adaptive quality using up to 50% less compute, or improve quality by up to 10% at fixed budget. In their routing setup they match the expensive decoder's performance while calling it only 50–75% of the time. Snell et al.'s 4× efficiency gain over best-of-N is the same idea arrived at from a different direction: their compute-optimal strategy is exactly a per-prompt allocation rule.

What to carry away

Six things, in rough order of how often I see them missed.

One. Coverage and accuracy are different quantities with different scaling behaviour. Coverage keeps rising for a very long time; the ability to identify which sample is correct usually does not. Any claim about test-time scaling should say which of the two it measured. pass@k is a capability measurement, not a deployable metric, unless you have a verifier that makes them equal.

Two. The smooth population-level scaling curve is produced by heterogeneity in problem difficulty, not by any single problem scaling smoothly. This is why the curve's exponent is a property of your benchmark as much as of your model, and why difficulty stratification should be table stakes in reporting.

Three. Verification is the binding constraint. With a perfect verifier, sampling scales beautifully; with a false-positive rate φ, accuracy is capped at p/(p + (1−p)φ) regardless of budget, and the cap tightens as problems get harder. Spending on a better verifier usually dominates spending on more samples.

Four. Serial and parallel compute are not interchangeable. They cost the same in weight FLOPs and KV bytes but differ by a factor of K in attention FLOPs and in latency, and they fail differently — width gives you candidates without an opinion, depth gives you an opinion without alternatives.

Five. More thinking can hurt, for reasons that are mechanistic rather than accidental: distractor capture, framing overfit, drift away from a correct conclusion already reached, and amplification of procedures the training data never properly installed. The inverted U is real, and its peak is earlier for easier problems.

Six. The interesting frontier is allocation, not extension. The optimal budget is per-prompt, it goes to zero for problems the model cannot solve, and it peaks just above that give-up threshold. Every system that does this well is really solving a difficulty-estimation problem in disguise.

And one open question I do not think anyone has answered. All the allocation theory above assumes accuracy is concave in compute, which is what makes water-filling correct. The inverse-scaling results say accuracy is sometimes not monotone, let alone concave. A genuinely correct allocation theory for the regime where extra compute can subtract value has not, as far as I can tell, been written down. That gap is where I would look next.

References

Grouped by topic. Every entry links to its primary source.