Mixing Data Without Losing Capabilities

The whack-a-mole problem

You have a pretrained base model. You want it to write code, do arithmetic, hold a conversation, follow a fussy formatting instruction, call a tool with valid JSON, answer in Hindi, summarise a contract, and refuse the three per cent of requests it should refuse. You want one set of weights to do all of that.

So you fine-tune. You add a good maths corpus and maths gets better. You add a coding corpus and coding gets better. You add agent trajectories and tool use gets better — and then you run the full evaluation suite and maths has dropped four points, the model has started answering everything in bulleted lists, and its refusal rate on benign medical questions has doubled.

Nothing in that story is a bug. It is the expected behaviour of a single parameter vector being asked to satisfy several objectives at once. The uncomfortable part is that the losses on your training data all went down. Every skill you trained on improved on its own training distribution. The damage happened somewhere you were not looking.

This article is about the structure of that damage: what causes it, how to measure it, how to predict it before spending a training run, and what actually works to prevent it. We will build the picture from gradients upward, because "data mixture" turns out to be a slightly misleading name. A data mixture is not really a recipe for a dataset. It is a knob on the distribution of gradients your optimiser sees, and every consequence follows from that.

Two things in that figure deserve to be pulled apart immediately, because they look identical on a dashboard and have completely different fixes.

Two failures wearing one name

"Catastrophic forgetting" has become the phrase people reach for whenever a number goes down. It covers at most half of what actually happens, and reaching for it in the wrong half sends you looking for the wrong fix.

Negative transfer

Suppose you train on everything at once. Your corpus is a union,

and the thing you actually minimise is a weighted sum of per-task risks:

Negative transfer is when pushing L_m down forces L_c up. The maths data is still in every batch. Nothing has been forgotten. The two tasks simply want the parameters to be in different places, and the optimiser is standing between them making a trade. Turn up λ_m and the trade moves. The failure is a property of the objective, and it is present from step one.

Catastrophic forgetting

Now train the same corpora in sequence: general instructions, then maths, then tools. Formally you are solving a different problem at each stage,

The optimiser has no incentive whatsoever to keep L_k−1 low. That it stays low at all is an accident of initialisation and of not moving very far. Push hard enough, or long enough, and there is nothing holding the old skill in place. The failure is a property of the path, and it grows with training duration.

These are genuinely different. Negative transfer is a bargaining problem — you can move the settlement point by changing weights. Forgetting is an erosion problem — no weight change fixes it, because the old loss is not in the objective at all. The standard fix for forgetting (put the old data back in the batch) works precisely by converting it into a bargaining problem, which you then have to negotiate.

A third thing that is neither

There is a failure mode that looks like forgetting on a benchmark and is not. Train heavily on one response style — terse code completions, say, or chain-of-thought maths — and the model starts producing that style everywhere. Its multiple-choice accuracy falls because it now writes a paragraph before answering and the parser takes the first token. Its refusal rate changes because refusals were phrased differently in the new data.

Nothing was erased. The readout moved. Zheng and colleagues call this spurious forgetting, and show that it is a loss of task alignment rather than of knowledge: the representations survive, the mapping from representation to output convention does not, and a few hundred examples restore it. Their controlled experiments find that early optimisation steps on a new task disrupt previously established task alignments, and they connect these shifts to orthogonal updates in the weights, then show that freezing the bottom layers substantially improves four continual-learning scenarios. We return to this in §11, because it changes what your mitigation budget should be spent on.

Why tasks interfere

Everything above was geometry-by-picture. Here is the actual mechanism, and it takes four lines.

Write the per-task gradient at the current parameters as

Take one gradient step on task A alone, θ⁺ = θ − η g_A, and ask what happened to task B. Taylor-expand L_B at θ:

so to first order,

That is the whole story of transfer, compressed. If ⟨g_A, g_B⟩ > 0 the two tasks want the same thing locally and training A is free improvement on B — positive transfer. If ⟨g_A, g_B⟩ < 0 the gradients disagree and every step you take on A actively damages B. And if the inner product is zero, the tasks are locally invisible to each other: not synergistic, not conflicting, just orthogonal.

The gradient Gram matrix

Now do the real thing. You are not training on one task; you are training on a weighted mixture, so the update direction is g = Σ_i λ_i g_i and the first-order change in every task's loss is

This is worth sitting with. The effect of a mixture on your capability vector is, locally, a linear map applied to the mixture weights. The diagonal of K is ‖g_i‖², always positive: every task helps itself. The off-diagonal entries are where all the drama lives. And because K is positive semi-definite, λ^⊤K λ ≥ 0 — the sum of losses always goes down. Individual losses are free to go up, and do.

Keep this matrix in mind. In §8 we will meet its empirical, finite-step, thoroughly non-linear cousin: the capability transfer matrix you actually estimate from ablation runs. K is what that matrix looks like when you squint at a single step.

Two things the inner product does not capture

Magnitude imbalance. Suppose ⟨g_A, g_B⟩ > 0 but ‖g_A‖ = 20‖g_B‖. The sum g_A + g_B points almost exactly along g_A. Task B is not being harmed; it is being ignored, which over a long run is indistinguishable. This is the motivation for GradNorm, which adaptively tunes per-task loss weights so that the gradient magnitudes of different tasks stay comparable and no task's learning is drowned out.

Curvature. The second-order term (η²/2) g_A^⊤H_Bg_A is non-negative wherever B's loss is locally convex, and it grows quadratically in the step size. Two tasks whose gradients agree can still hurt each other if you step far enough that you sail past B's valley floor. Yu and colleagues named the joint condition — conflicting gradients, high positive curvature, and large differences in gradient magnitude — the tragic triad, and proposed PCGrad: when two task gradients conflict, project one onto the normal plane of the other,

It is a satisfying geometric idea. The authors identified conflicting gradients, high positive curvature and large gradient differences as the conditions underlying the major challenges in multi-task optimisation, and showed substantial improvements across supervised and reinforcement-learning benchmarks.

A mixture is a loss weighting

Here is the sentence that reorganises everything. In §2 I wrote a weighted objective with explicit λ's and noted that they need not appear anywhere in your config. That is because you already set them, when you decided how much of each dataset to put in.

Let each source D_i be sampled with probability p_i, and within a source let examples be uniform. Then the training objective is

and by linearity the same holds for the gradient the optimiser actually receives:

This is why the framing at the top of the article — a mixture is a knob on the distribution of gradients — is literal rather than metaphorical. Combining it with the Gram matrix from §3 gives the local law of post-training data mixing:

Three ways the identity leaks

Tokens, not examples. Almost every LLM trainer averages the loss over tokens in a packed batch, not over examples. If source i has mean supervised length ℓ̄_i, the weight that actually reaches the gradient is

This is the single most common way a carefully designed mixture turns into a different one in practice. It is also why serious post-training reports show their mixture broken down by token count and not only by example count — Tülu 3's mixture figure plots sources against prompt-plus-completion length for exactly this reason. If your trainer instead normalises per example (mean-of-per-example-means), you get the nominal weights back but you have quietly down-weighted every token of every long response.

Mean is not everything. Two mixtures can have identical p and very different gradient noise. Sampling 10% from a pool of 5,000 examples for a long run means seeing each of them dozens of times; sampling 10% from a pool of two million means never seeing one twice. The expected gradient is the same. The variance, the overfitting behaviour, and the effective number of distinct gradients are not.

Loss weight is not learning rate. A task that is already nearly solved has a small gradient. Doubling its mixture weight doubles a small number. Meanwhile a noisy, hard, or badly formatted source can have a large gradient at 5% and dominate the update. Mixture weight buys you attention from the optimiser; how much progress that attention converts into depends on how much loss is left and how well-conditioned the source is.

The naive mixtures

Why the field moved beyond one fixed ratio. Size is not value, equal exposure is not equal learning, and transfer changes over the run. Each newer method adds information—caps, proxy evaluations, gradients, or training phase—to repair one missing assumption.

You have K sources of wildly different sizes. Before any clever method, there are three obvious answers, and it is worth being precise about how each one fails, because every sophisticated method is a repair of one of them.

Dataset-proportional

Concatenate everything and shuffle. This has one genuinely lovely property that people forget: every source is seen the same number of times. If you train for B examples, source i is seen B p_i / |D_i| = B / Σ_j|D_j| times — independent of i. No source is memorised more than any other.

The failure is that dataset size is a fact about scraping, not about importance. A 2M-example synthetic code corpus was easy to generate; a 900-example set of carefully written refusals was not. Proportional mixing hands the run to whoever had the cheapest data pipeline.

Equal per task

Now importance is decoupled from size — but the epoch count is not. Under a budget of B examples, source i is seen B/(K|D_i|) times, which for a small source is enormous. A 700-example task inside a 24-task equal mixture with a million-example budget is trained for roughly sixty epochs. It will be memorised verbatim, its gradient will collapse toward zero on its own data while still contributing noise to everyone else's, and any labelling error in it will be learned as a rule.

Temperature sampling

The standard interpolation, borrowed from multilingual translation and multilingual pretraining, is to raise sizes to a power:

It helps to have a single number for "how many sources is this mixture really training on". The natural one is the perplexity of p:

Sweeping α from 1 to 0 walks N_eff from "three sources own the run" up to K, while walking the maximum epoch count from below one up into the dozens. That is the entire trade, and it is a trade between two different failure modes rather than between good and bad.

Capping instead of tempering

The other classical answer, used in T5's multi-task mixtures and inherited by the FLAN collections, is examples-proportional mixing with an artificial limit:

This is a piecewise version of the same idea and it has a nicer failure mode: it never up-samples a small source at all, it only stops large ones from running away. If your worry is memorising the tail, capping is strictly safer than tempering. If your worry is that the tail never gets learned, capping does nothing for you and tempering is the tool.

Or: throw some tasks away

The assumption behind all of the above is that every source should get some weight. It is not obvious. Renduchintala and colleagues framed mixture design for instruction tuning as a submodular budget-allocation problem — assign diminishing-returns importance scores to tasks and select non-redundant examples within them — and reported that this significantly outperforms both examples-proportional and equal mixing, and that in a limited-budget setting, allocating the budget among a subset of representative tasks beats distributing it among all tasks.

That last clause is the interesting one. Under a fixed budget, pruning tasks entirely can beat including them at low weight. A task at 0.4% weight contributes almost nothing to its own capability while still contributing its full share of conflicting gradients and its full share of stylistic contamination. Below some threshold, a source is all cost and no benefit.

Instruction tuning was always multi-task

It is easy to read "post-training data mixture" as a recent concern. It is not. Instruction tuning was born as a multi-task learning method, and its entire history is a history of people discovering — repeatedly, and usually the hard way — that how you balance the tasks matters as much as which tasks you have.

The arc runs roughly like this. Single-task fine-tuning gives one model per task. Multi-task fine-tuning shares a backbone across tasks and needs a mixing rule, which is where examples-proportional-with-a-cap came from. Instruction tuning reframes each task as a natural-language instruction, which turns the task list into a training distribution over formats as well as contents, and — the surprising part — buys generalisation to tasks that were never in the mixture at all. General post-training then drops the notion of a task list entirely and designs against a list of capabilities.

What the progression actually taught

Why the field moved this way. Task counts stopped being useful once differently named datasets began teaching the same capability. The unit of design therefore moved upward—from examples, to tasks, to prompt formats, and finally to skills. This also explains why “train only on what you evaluate” fails: complementary formats can create positive transfer, while safety or specialist data may be almost orthogonal and must be measured on separate axes.

What are we actually optimising?

Everything so far assumed we knew what "better" meant. We do not, and the choice matters more than most mixture methods do.

The default is to average the evaluation suite. Consider two candidate models:

By the mean they are the same model. By any product judgement they are not remotely the same model: A cannot call a tool, so any workflow that needs one fails completely, and no amount of excellent chat compensates. The average is a bad objective because capabilities compose conjunctively in use — a task needs the model to be adequate at all of the things it touches — while the average scores them disjunctively.

Four objectives worth naming

Form (4) is the one people are living but almost never write down. Writing it down changes decisions: it makes explicit that ε is a policy choice (is a two-point drop in maths acceptable to gain thirty points of tool use?), and it makes the release criterion checkable rather than argued.

Why max-min is not just a fussier average

There is a real geometric reason to care about the difference. Suppose you optimise a linear objective Σ_i u_iS_i over the achievable set of capability vectors. The maximiser of a linear function over a set always lies on the boundary of that set's convex hull. If the Pareto front is non-convex — if it dips inward in the middle — then the balanced solutions in that dip are unreachable by any choice of weights. You can sweep u across the whole simplex and you will jump from one extreme to the other, never landing in between.

Max-min (and its smoother relatives, the Chebyshev scalarisations) does not have this limitation, because min_iS_i is not linear. If you have ever swept a mixture weight and watched a capability flip discontinuously from "fine" to "broken" with nothing sensible in between, a non-convex frontier is one honest explanation.

The objective is not always monotone

Every formulation above quietly assumes that more of a capability is better. For safety that is false in a way that has been measured. Bianchi and colleagues found that adding just 3% safety examples — a few hundred demonstrations — substantially improves the safety of a LLaMA-style model without significantly reducing capability or helpfulness on standard benchmarks, but that too much safety tuning produces exaggerated safety behaviours, where models refuse perfectly safe prompts that superficially resemble unsafe ones.

So S_safety(w) as a single number is not a thing you can maximise. What exists is a pair — attack success rate, and over-refusal rate on benign lookalikes — moving in the same direction as you add data. The correct objective is two-sided, and the correct mixture weight is an interior point. Three per cent, not thirty. This generalises: any capability defined by a threshold behaviour (refuse / don't refuse, call a tool / answer directly, cite / don't cite) has a lookalike failure mode on the other side, and its mixture weight is a hump rather than a ramp.

The transfer matrix

The most useful object in post-training data design is a matrix that almost nobody writes down explicitly, even though everybody's ablation tables are estimating it one cell at a time.

Rows are training sources. Columns are evaluated capabilities. This is the empirical, finite-step, thoroughly non-linear descendant of the gradient Gram matrix from §3: K tells you what happens to each task's loss after one step; M tells you what happens to each benchmark after a whole run. Same idea, different regime, and the differences matter.

Note what the definition buys you. Under a first-order model, the capability vector you get from a mixture w is

and every objective from §7 becomes a small optimisation problem you can solve on a laptop: linear programming for the weighted objective, a linear program with an epigraph variable for max-min, and a linear program with capability floors for the constrained form. The expensive part is M, and it is expensive exactly once.

How you actually estimate it

Three designs, in increasing order of cost and usefulness.

Leave-one-out ablation. Train the full mixture, then retrain K times with one source removed each time. The change in each capability gives you a column of effects for that source. This is exactly what Tülu 3's ablation table does — remove safety, remove diverse chat, remove persona data, remove maths, and read off what moved. Cost: K+1 runs. Gives you signs and rough magnitudes at your current operating point, which is often all you need.

Add-one-in. Start from a minimal base mixture and add each source alone. Cost: also K+1 runs, and it answers a different question — the marginal value of a source in isolation rather than at the margin of a full mixture. These two answers can differ enormously when sources are redundant. Two overlapping maths corpora each look valuable added alone and each look worthless removed from the full mix.

Randomised designs. Sample many random mixtures on the simplex, train small proxy models on each, and fit a response surface. Cost: dozens of small runs, but you get a function rather than a table, and you can optimise it. This is §9's territory.

Four properties that break your intuition

It is not symmetric. K was symmetric because it was a Gram matrix. M is not even square in general — its rows are indexed by training corpora and its columns by evaluation suites, which are different sets of things. Even when they are named the same, "does code data help maths" and "does maths data help code" are separate empirical questions with, in practice, different answers.

It is not constant. M depends on the base model, on the total budget, and — most annoyingly — on the rest of the mixture. The derivative ∂S_j/∂w_i is evaluated at a point, and the point moves.

Its off-diagonals change sign with budget. Dong and colleagues studied maths, code and general-alignment composition in SFT and found that data composition enhances various abilities under limited data conditions, yet can lead to performance conflicts when data is plentiful, and that the amount of composition data influences performance more than the composition ratio. Read that as a statement about M: the off-diagonal entries are positive in the small-data regime and negative in the large-data regime. Which means an ablation run at 50k examples can give you the opposite sign from the same ablation at 500k, and you will trust the wrong one.

There are interaction terms, and some are large. The linear model S ≈ S⁽⁰⁾ + M^⊤w assumes sources combine additively. The FLAN Collection's zero-shot / few-shot result is a flat contradiction of that: mixing the two formats beats either alone in both evaluation settings, which is a positive second-order term. So the honest model is

The practical compromise is to fit the linear model everywhere and the quadratic terms only for the handful of pairs you suspect interact: format pairs (zero-shot × few-shot, with-CoT × without-CoT), redundancy pairs (two corpora covering the same skill), and the pairs your ablations show behaving inconsistently.

Mixture laws

Grid search is the honest baseline and it is unaffordable. With K sources at m levels each you have m^K−1 mixtures; at nine sources and five levels that is 390,625 training runs. Nobody has ever done this and nobody ever will.

The alternative that has emerged over the last two years is to treat the mixture-to-performance map as a function to be fitted rather than a space to be searched. Train a modest number of small proxy models on scattered mixtures, fit a parametric law, optimise the law analytically, and spend your one real run on the answer.

This is a direct descendant of the compute-scaling laws in the parameters/tokens/compute piece, and the pretraining-side version of it appears in how LLMs are actually pretrained. What is new here is that the response variable is a capability vector, not a scalar loss, and that the constraint structure from §7 comes along for the ride.

What the literature has established

The functional form works. Ye and colleagues proposed data mixing laws with an exponential dependence on domain weights, fitted at small scale and extrapolated to larger models and data budgets via the standard power laws. Apple's group generalised this: their scaling laws predict the loss of a model of size N trained on D tokens with a specific domain-weight vector h, and they validate the universality of this across LLM, native multimodal and large vision pretraining, showing that the parameters can be estimated from a few small-scale runs and used to estimate performance at larger scales and unseen domain weights.

It transfers to SFT. The most directly relevant result for post-training frames SFT mixture selection as an optimisation problem: the approach parametrises the loss by modelling effective data transferred and leveraging scaling laws for fine-tuning, fits the parameters from small-scale mixture experiments, and derives optimal weights — with per-domain loss only 0.66% higher on average than the best domain loss found by grid search. That number is the point of the whole enterprise: a fitted surface got within a fraction of a per cent of exhaustive search, at a tiny fraction of the compute.

Proxy-model methods are the other branch. DoReMi trains a small proxy with group distributionally-robust optimisation to produce domain weights, then trains the large model with them. RegMix fits a regressor over many tiny runs and selects the top-predicted mixtures. Both trade a fitted parametric law for a learned one.

The gap nobody has closed

Every one of these laws predicts loss. What you ship is judged on capabilities: pass@1 on a code benchmark, exact match on a maths set, win rate against a reference model, attack success rate. The relationship between per-domain validation loss and any of those is monotone-ish, noisy, and occasionally non-monotone — a model can lower its loss on a maths corpus by getting better at imitating that corpus's formatting while getting no better at maths.

In practice this means the fitted law is a good proposal mechanism and a bad acceptance mechanism. Use it to generate a handful of candidate mixtures, then evaluate those candidates on the real suite. Optimising a loss surface all the way to its argmin and shipping the result is how you end up with a model that is excellent at your validation set's writing style.

Static mixtures and curricula

Nothing we have written requires w to be constant. The objective is a function of the whole trajectory, so the natural object is a schedule w(t), and the static mixture is the degenerate case.

There are three reasonable arguments for making it move.

Formats before skills. A base model does not know what an assistant turn looks like. Early in post-training, almost any instruction data buys a large improvement, because the model is learning the shape of the interaction rather than any particular capability. Once that is in place, the marginal value of more generic instruction data collapses — which is exactly what Dong and colleagues measured: general abilities plateau after roughly a thousand samples, while mathematical reasoning and code generation keep improving with more data. A schedule that spends its first few thousand steps on format and its remaining budget on hard skills is following that curve.

The learning rate is not constant either. Whatever the model sees while the learning rate is still high gets partially overwritten; whatever it sees at the end, under a decayed learning rate, sticks. This is the same reasoning behind late-stage high-quality data in pretraining, and it applies unchanged. A static mixture already has an implicit curriculum — it just happens to be the shuffle order.

Feedback. If a capability is lagging at step 40,000, why keep the weight you chose at step zero? Online methods estimate the mixing law during training and adjust proportions — Aioli's contribution is precisely to do this with parameter estimates that are actually calibrated, and it is the version of dynamic mixing with the strongest evidence behind it.

The loss-chasing trap

The obvious dynamic rule is: up-weight whatever has the highest loss. It is wrong, and the reason is worth stating precisely.

Decompose a source's loss into a reducible and an irreducible part:

A source with mislabelled targets, contradictory annotations, or intrinsically high-entropy outputs (open-ended creative writing, for instance) has a large L_i^∞. A loss-chasing controller sees a persistently high number, concludes the model is failing there, and pours budget in. The loss does not fall, because it cannot. The controller pours in more. Meanwhile every other capability is being starved by a source that is, by construction, unlearnable.

The fix is to allocate on the derivative, not the level — up-weight sources whose loss is still falling fastest per unit of budget:

which is exactly what a fitted mixing law gives you and what a naive loss-balancing heuristic does not.

What forgetting actually is

Two facts about forgetting in language models are worth having in your head before you spend budget preventing it.

The first is that it is real and it does not politely shrink with scale. Luo and colleagues evaluated continual instruction tuning across domain knowledge, reasoning and reading comprehension, and found that catastrophic forgetting is generally observed in models from 1B to 7B parameters, and that as model scale increases the severity of forgetting intensifies in that range — which they attribute partly to the larger model's higher initial performance, i.e. it has more to lose. They also found that general instruction tuning beforehand helps alleviate forgetting during subsequent fine-tuning — a strong argument for the format-first curriculum in §10.

The second is that a large fraction of what gets counted as forgetting is not loss of knowledge at all. This is the spurious-forgetting result from §2, and it has a sharp operational consequence: diagnose before you mitigate. Replay, regularisation and merging are all expensive. Restoring an output convention is not.

The four rungs

Rung 1 — change the prompt, not the model. Re-run the evaluation with a different answer format, a different extraction regex, or two few-shot exemplars in the old style. If the metric comes back, nothing was forgotten; the model changed its output convention and your harness could not follow. This costs an evaluation run.

Rung 2 — a tiny re-alignment set. Fine-tune on 50–200 examples of the old task's format. Spurious forgetting is reversible at roughly this scale, because the representations were never damaged — only the mapping from them to the output convention. If two hundred examples restore a capability that a hundred thousand examples appeared to destroy, the destruction was superficial.

Rung 3 — probe the representation directly. Ask whether the information is still linearly decodable from hidden states, or whether the model gets the answer right when the task is posed as multiple choice rather than free generation. Knowledge that is present but not expressible is an alignment problem; knowledge that is absent under every probe is an erasure problem.

Rung 4 — measure the relearning cost. If you must retrain, measure how many examples it takes to get back to the original score. The unlearning literature makes this the central metric for a reason: "forgotten" knowledge can often be rapidly recovered through minimal fine-tuning even on unrelated data, and distinguishing reversible from irreversible forgetting requires looking at representations rather than task-level accuracy. A capability that takes 500 examples to restore was displaced. One that takes 50,000 was destroyed. Only the second justifies a replay budget.

Why the mechanism makes sense

The spurious-forgetting analysis connects the effect to orthogonal updates in the weights early in a new task, and finds that freezing the bottom layers mitigates it. That is coherent with a simple picture: the lower layers build features that many tasks share, and the upper layers implement a task-conditional readout. A new task's first few hundred steps mostly rewrite the readout — cheap to learn, cheap to destroy, cheap to restore. Sustained training on a very different distribution eventually reaches down and rewrites the features, which is the expensive kind of damage.

This also explains the scale finding. A larger model has a sharper, better-fitted readout to disrupt, and starts from a higher score, so the measured drop is larger even when the underlying features are equally intact.

The antidote toolkit

Suppose the diagnosis came back as genuine interference or genuine erasure. There are seven families of fix, and they are not interchangeable — each one buys a specific thing at a specific price.

1. Replay

Instead of training on the new data alone, train on

Replay converts a forgetting problem into a negative-transfer problem, which you then negotiate with weights — exactly the reframing from §2. Ibrahim and colleagues studied replay rates of 1%, 5%, 10%, 25% and 50% across a weak (English→English) and a strong (English→German) distribution shift and found that replay significantly reduces forgetting across both shifts, and that the stronger shift requires more replay to mitigate forgetting. Combined with learning-rate re-warming and re-decaying, this was sufficient to match the performance of fully re-training from scratch on all available data, at a fraction of the compute.

The subtlety is that replay is compute-equivalent: every replay token displaces a new-data token. Adding replay is not free retention, it is retention bought with adaptation.

Two practical notes. First, synthetic replay — generating the replay set from the model you are about to modify — is targeted in a way that archival replay is not: you can steer generation toward precisely the behaviours you want to preserve, and you do not need the original training data to still exist. Second, replay composition matters as much as replay quantity. AgentBank's ablation found that a relatively low proportion of generalist data improves agent performance on unseen tasks, but as generalist data continues to increase, held-out performance degrades sharply — replay has an interior optimum too, and the same is true of AgentTuning's hybrid strategy of mixing agent trajectories with general-domain instructions to acquire agent skills without compromising general abilities.

2. Regularisation

Penalise moving away from the previous solution, either uniformly (‖θ − θ₀‖², i.e. L2-to-init) or per-parameter with an importance weight (elastic weight consolidation, where the weight is the diagonal Fisher information from the old task). Cheap, principled, and in practice weaker than replay for LLMs — because a language model's capabilities are not localised in a small set of parameters the Fisher can identify.

3. Parameter-efficient isolation

Train an adapter and leave the base weights alone. Biderman and colleagues ran the careful comparison on code and maths and found the trade-off is real in both directions: in standard low-rank settings LoRA substantially underperforms full fine-tuning, but it better maintains the base model's performance on tasks outside the target domain, and it mitigates forgetting more than weight decay or dropout. The magnitudes are concrete — in code instruction-tuning, full fine-tuning drops to 0.414 on the forgetting metric versus 0.509 for LoRA at rank 64, and forgetting worsens with training duration. They also report that full fine-tuning learns perturbations of 10–100× higher rank than typical LoRA configurations, which is a satisfying explanation for both halves of the title.

4. Model merging

Train specialists separately, then combine them in weight space. Task arithmetic defines a task vector τ_t = θ_t − θ₀ and merges by adding scaled task vectors back to the base. The failure mode is interference between the vectors, which TIES attacks in three steps: trim low-magnitude parameters back to their pretrained values, elect a consensus sign per parameter across tasks, and merge only the entries that agree with it; DARE instead randomly drops a large fraction of delta parameters and rescales the survivors.

The appeal for post-training is real: specialists never interfere during training, so you get each capability at close to its ceiling, and you pay the interference cost once, cheaply, at merge time. Tülu 3 explicitly explored merging alongside mixture ablations for exactly this reason. The catch is that merging quality degrades as you add models, sign conflicts grow with the number of tasks, and you still need a held-out suite to choose the merge coefficients — which is a mixture search wearing a different hat.

5. Reduce the sharing

All interference in §3 came from tasks sharing parameters. Mixture-of-experts routing lets different tasks use different subsets of the network, which mechanically lowers the off-diagonal terms. This is a real architectural answer, it is expensive to adopt mid-project, and it trades interference for a load-balancing problem.

6. Gradient surgery

Covered in §3. Correct in principle, impractical at scale, and with a strong sceptical baseline.

7. Just fix the data

Listed last because it is chronically underrated. A large share of apparent interference is contamination: a code corpus whose responses always begin with a preamble teaches that preamble to every task; a maths corpus with chain-of-thought in a rigid template teaches that template as the answer format; a safety set written in one voice teaches that voice. These look like capability conflicts on a dashboard and are in fact style transfer. Normalising response format across sources removes a surprising amount of "interference" for the price of a preprocessing script.

Quality, diversity, and the long tail

Everything so far has treated a source as a scalar: a weight p_i on a pool. That is a useful abstraction and it is wrong in three specific ways. Two examples from the same source are not interchangeable; the second thousand examples from a source are worth far less than the first thousand; and a small number of examples inside a source are actively harmful.

Diminishing returns are per-capability, not global

The single most useful empirical fact in this section comes from Dong and colleagues' study of SFT composition. Alongside the composition finding, they report that general abilities such as reasoning and reading comprehension can be enhanced with roughly a thousand samples and improve little beyond that, whereas mathematical reasoning and code generation improve consistently as data volume grows.

Two capabilities in the same mixture with entirely different saturation curves. Which means a uniform weight is wrong almost by construction: the right allocation gives general chat its first thousand examples and then stops, and keeps feeding maths and code. This is the same statement as the reducible-loss rule in §10, arrived at from data rather than from dynamics — and it is why "we doubled the chat data and nothing happened" is such a common experience.

Diversity is the thing that is actually scarce

What the saturation curves are really measuring is coverage. The first thousand chat examples teach a format, a register, a refusal style, a way of handling ambiguity. The next fifty thousand mostly restate them. The curve flattens not because the model has learned everything about chat but because your corpus has run out of new things to say.

This reframes deduplication. Removing near-duplicates is usually sold as a hygiene measure; it is better understood as an accounting correction. A source with 40% near-duplicates has an effective size of 60% of its nominal size, and every mixture weight you set on it is silently 40% smaller than you think. Since §4 showed that weights are already distorted by token length, this is the second systematic gap between the number in the config and the number the optimiser sees.

The strongest form of this argument is LIMA, which showed that a small, carefully curated instruction set can produce a strong assistant, on the hypothesis that alignment is largely about surfacing formats and behaviours already present from pretraining rather than teaching new knowledge. Whether or not the strong form holds, the operational lesson is robust: when a source has saturated, additional examples from it are close to free in cost and close to worthless in value, and both halves of that sentence matter for budgeting.

Selection beats spreading

Given a fixed budget, is it better to take a few examples from every task or many examples from a few? Renduchintala and colleagues answered this directly with SMART, which uses submodular functions to allocate budget across tasks and then select representative instances within them. They found their method outperforms both examples-proportional mixing and equal mixing, and that under a limited budget it is more effective to allocate the budget to a subset of representative tasks than to spread it across all of them.

The mechanism is exactly §5's effective-source-count argument. Ten examples from a task teach the model that the task exists without teaching it how to do the task; a hundred tasks at ten examples each spends the whole budget on establishing existence. Coverage of the task space comes from tasks being representative, not from tasks being numerous.

The examples that are worse than useless

Three categories deserve a dedicated pass, because their damage is not proportional to their frequency.

Contamination. Tülu 3's decontamination pipeline is a good concrete recipe: 8-gram matching between training instances and test sets, counting a match when more than 50% of a test instance's tokens appear in an 8-gram overlap, and removing any training dataset with more than 2% contaminated instances against their development and unseen evaluation suites. Note the shape: a per-instance match rule, a per-dataset drop threshold, and it is run against the evaluations that decide the mixture. Contaminated data does not just inflate a score; it corrupts every ablation in §8, because the transfer matrix is estimated from those scores.

Wrong answers with confident formatting. A maths example with a flawless chain of thought and a wrong final answer teaches the format and the error together. Because format is the easier thing to learn, these examples are disproportionately effective at teaching the wrong lesson. Filtering by verifiable correctness where correctness is checkable — and it is checkable for a large fraction of maths and code — is the highest-value filter available.

Style monoculture inside a source. Covered in §12, mentioned again because it is the most common cause of what looks like negative transfer. If every example from a source begins with the same preamble, the model learns the preamble as part of the task and exports it everywhere.

A protocol that survives contact

Here is the whole article compressed into an order of operations. It is deliberately front-loaded with cheap steps, because most of the value is in the first three and most of the compute is in the last two.

Step 0 — Write the objective down

Before touching data, write the sentence that a mixture is allowed to be judged by. "Maximise the average of eight benchmarks" and "maximise the average subject to safety ≥ 0.95 and no capability falling more than two points below the current model" are different problems with different answers (§7). If you cannot write the sentence, the search has no objective and every result will look defensible.

Include the noise floor. If your evaluation has a standard error of one point, differences under two points are not results, and a search that chases them is fitting noise.

Step 1 — Fix the data before weighting it

Deduplicate, decontaminate against the deciding evaluations, normalise response format, filter for verifiable correctness where possible, and record the effective size and mean token length of each source (§4, §13). Every downstream number depends on this being done first.

Step 2 — Run stratified sampling and record it as the number to beat

Equal or capped-proportional weights, one run, full evaluation suite. This is the baseline that a large fraction of published mixture-optimisation methods fail to beat (§9), and any method you adopt has to clear it by more than the noise floor. Skipping this step is how teams end up believing a complicated method works.

Step 3 — Build a small transfer matrix

Leave-one-source-out ablations at the smallest scale that preserves your sign structure, evaluated on the full suite (§8). Read the rows to find sources that are pulling their weight, the columns to find capabilities that regress, and the asymmetries to find one-directional dependencies. Do this at two budgets if you can afford it, because off-diagonal signs flip with budget and one measurement will mislead you.

Step 4 — Handle the exceptions explicitly

Safety is a hump, not a ramp: enough to move the metric, not enough to induce over-refusal, and evaluated on both harmful prompts and benign lookalikes (§7). Sources that are orthogonal to everything else — safety often is — should be set from their own curve rather than folded into the global search, because the search will not find their optimum and does not need to.

Step 5 — Only now, search

Fit a mixture law on a modest number of proxy runs and optimise the fitted surface (§9). Sample the simplex with a low-discrepancy design rather than a grid. Verify the fit on held-out mixtures, not on the ones it was fitted to, and treat the fitted optimum as a candidate to be confirmed by a full run — not as an answer.

Step 6 — Choose the schedule, not just the weights

Decide whether format-first ordering, an annealing tail, or a static mixture is right for your run (§10). Do not adopt a dynamic method without first establishing that a static mixture at the same average weights is worse; loss-chasing schedulers have a well-defined failure mode and the check is one extra run.

Step 7 — Diagnose regressions before paying for them

When something drops, walk the ladder in §11 before reaching for replay. Format probe, small re-alignment set, representation probe, relearning cost. Three of the four are cheaper than the mitigation you were about to apply.

Step 8 — Keep the ledger

Record, per run: the mixture weights, the effective source sizes, the mean token lengths, the resulting token-level shares, and the full evaluation vector with error bars. Six months later this table is the transfer matrix you wish you had, assembled for free. Teams that keep it can answer "what happens if we drop this source" without running anything; teams that do not, re-run the ablation every time.

What we still do not know

It would be tidy to end with a solved problem. This one is not solved, and the honest list of gaps is short enough to state plainly.

Whether mixture optimisation beats stratified sampling in general. The Aioli result is the uncomfortable one: a careful unification of the existing methods found that none of them consistently outperforms simple stratified sampling on average test perplexity per group, and located the failure not in the functional forms but in how their parameters are estimated. The SFT-side results are more encouraging — a fitted law landing within 0.66% of grid search is a real result — but the field does not yet have a method you can adopt without checking it against the boring baseline on your own data.

What the right unit of a "task" is. Every method in §9 optimises weights over a partition of the data, and every result depends on that partition. Coarse groupings hide interference inside a group; fine groupings make the search intractable and the estimates noisy. Nobody has a principled way to choose the granularity, and the effect of choosing wrong is not usually measured.

How transfer behaves across scale. Mixture laws assume the domain-weight dependence extrapolates while the size dependence is handled by standard power laws. That assumption has been validated in pretraining, including across modalities. It is far less validated in post-training, where the relevant capabilities are latent in the base model and the mixture is mostly deciding which of them get surfaced. A transfer matrix measured at 1B may not have the same sign structure at 70B — and §8's budget-dependent sign flips are a warning that it might not have the same signs at the same scale with a different budget either.

How much post-training forgetting is real. Spurious forgetting says a substantial share of measured drops are alignment disruption rather than knowledge loss. Nobody has a clean estimate of the share. This matters directly for budgeting: if most measured forgetting is superficial, the field is over-spending on replay and under-spending on format discipline.

Whether specialist ceilings are a real constraint or a search failure. The generalist gap in §1 is consistently observed and rarely explained. It could be an irreducible consequence of finite capacity and conflicting gradients, or it could be that nobody has searched the mixture space well enough. The gap has narrowed as mixtures have got better, which is weak evidence for the second.

Whether the objective is measurable at all. Every method here optimises a vector of benchmark scores. Benchmarks are noisy, saturating, contaminated with varying severity, and correlated with each other in ways nobody enumerates. A mixture search is a search over a proxy, and the gap between the proxy and the thing you want is not something more compute fixes.

The systems side of all of this — how a mixture actually becomes batches, what the data loader does with per-source weights, and where the throughput goes — is the subject of Inside a Training Step. The scaling side, and where the token budget in §13 comes from, is in Scaling Laws from First Principles.

References

Grouped by the argument they support. Every title links to the primary source; where a paper appears in more than one section of the article it is listed once, under the group where it does the most work.