Teaching a Language Model to Follow Instructions

The gap a base model leaves

Pretraining ends with a model that is extraordinarily good at one thing: guessing what comes next in a document drawn from the internet. That is not the same thing as answering you.

The previous article in this series followed a corpus from raw crawl through deduplication, filtering and decontamination into a foundation model. At the end of it you have a network that has consumed something on the order of ten trillion tokens and has, in a real sense, absorbed an enormous amount of the world. Ask it about Bayes' rule, Bengali grammar, or the failure modes of a distributed all-reduce, and the knowledge is in there.

But consider what actually happens when you type a question into a raw base model. The objective it was trained on says nothing about questions and answers. It says: given this prefix, what token is statistically likely to follow, in text that looks like the pretraining corpus. And in the pretraining corpus, a line that reads "What is the capital of France?" is very often followed not by "Paris" but by another question, or by a multiple-choice stem, or by an answer key three paragraphs down, or by the boilerplate of a quiz site.

The base model is not being unhelpful. It is being faithful. It has correctly modelled a distribution in which your prompt is a fragment of a document, not a turn in a conversation.

Three things are missing, not one

It is tempting to summarise the gap as "the model doesn't know it should answer." That is one part of it, and the least interesting part. Pull it apart and there are at least four distinct deficits, which matters because they are fixed by different things and they fail in different ways:

SFT is the tool for the first two and a large part of the third. It is a poor tool for the fourth, for reasons that will become mechanical rather than philosophical by the end of §3, and that is precisely why the pipeline does not stop here.

Same loss, different distribution

Start from the pretraining objective, unchanged. For a sequence x = (x₁, …, x_T) the model is trained to minimise the negative log-likelihood of each token given its prefix:

Averaged over a data distribution D, this is cross-entropy, and cross-entropy has a decomposition worth writing out because it makes the mechanism visible:

So minimising the loss means minimising KL(D ‖ p_θ): pushing the model's distribution toward the data's. Nothing in that statement mentions the internet. The gradient is completely indifferent to what D is. Swap it and you have swapped what the model becomes.

What that substitution looks like concretely

PRETRAINING — a fragment of a page

  The capital of France is Paris. Paris is located on the
  Seine, in the north of the country, and has been the seat
  of French government since ...

SFT — a conversation that should have happened

  <|system|>    You are a helpful assistant.
  <|user|>      What is the capital of France?
  <|assistant|> Paris.<|eot|>

Both are token sequences. Both are scored with the identical loss. The difference is entirely in what the token sequences are. In the first, "Paris" is a fact embedded in expository prose. In the second, "Paris" is a response, and the surrounding role markers make it structurally impossible to confuse the two.

That second point deserves emphasis because it is the seed of the entire preference-optimisation literature. Forward KL, KL(D ‖ p_θ), is heavily penalised when the model assigns near-zero probability to something the data does. It is barely penalised at all when the model assigns high probability to something the data never does. So SFT is a mode-covering, permissive objective. It teaches "do all of these things"; it has no vocabulary for "and do not do that." Reverse KL — which is what appears in the RLHF objective — has the opposite character. That asymmetry is most of the reason a second stage exists.

An analogy that holds up

Think of a session musician who has spent twenty years listening to every recording ever made. They can play anything in any style; they have the vocabulary. What they have never done is sit in your band and take a cue. The first rehearsal is not about teaching them music. It is about establishing that when you nod, they come in; when you raise a hand, they stop; that this room plays quietly.

That rehearsal is SFT. It is short relative to the twenty years. It does not add much to their vocabulary. It changes almost everything about what happens when you nod. And critically: if you only ever rehearse songs the band already plays well, you will find out nothing about what happens on the songs you have not rehearsed — which is exactly the failure mode we will formalise next.

SFT is behaviour cloning

There is a more useful frame than "next-token prediction on nicer data," and it is worth setting up carefully because it explains most of what goes wrong later.

Treat a conversation as a trajectory. The state s_t is everything generated so far — the system prompt, the user turn, and whatever assistant tokens have already been emitted. The action a_t is the next token. The policy is the model, π_θ(a_t | s_t) = p_θ(x_t | x_<t). There is an expert — a human annotator, or a stronger model, or a verifier-filtered sample from the model itself — whose policy π^* produced the demonstrations.

This is textbook behaviour cloning: fit a policy by maximum likelihood on expert state–action pairs. It is the oldest and simplest imitation-learning algorithm, and it has one famous, structural weakness.

The states you train on are not the states you visit

During training, every prefix the model conditions on comes from the demonstration. Token 40 is scored given the expert's tokens 1–39, not the model's. This is teacher forcing, and it is what makes the loss decomposable and the training embarrassingly parallel across positions.

During generation, the model conditions on its own tokens. Once it emits something the expert would not have, it is in a state that never appeared in the training distribution — and its behaviour there is whatever the network happens to extrapolate. If that state is also unusual, the next token is more likely to be off, and the state drifts further. Errors compound.

The classical analysis makes the shape of this precise. If a cloned policy makes a mistake with probability at most ε on states drawn from the expert's own distribution, then over a horizon of T steps the worst-case additional cost of the cloned policy relative to the expert grows like O(ε T²), not O(ε T). One factor of T from having T chances to err; a second factor because a single early mistake can put you in a bad region and keep you there for the rest of the rollout. Interactive methods that let the expert label states the learner visits — DAgger being the canonical one — recover the linear rate.

Four predictions this frame makes, all of which hold

Long outputs degrade faster than short ones. The T² term is not gentle. Assistants are noticeably more reliable in a two-sentence answer than in the fortieth paragraph of a long document, and the failure is not usually a wrong fact — it is losing the thread, repeating, or drifting in register.

Models are bad at recovering from their own mistakes. Demonstration data is overwhelmingly composed of clean trajectories: the expert did not make an error and then correct it. So there is no supervision for the state "I have just written something wrong." The model has never been shown what a competent recovery looks like from there. This is a data problem with a data fix — include recovery trajectories — and it is also the thing that long chain-of-thought training changes most dramatically, which we return to in §9.

Off-distribution prompts are where SFT models are weakest. Not because the knowledge is missing but because d_π* did not cover that region, so nothing constrained the behaviour there. Jailbreaks are, in this frame, a search for states the demonstration distribution did not reach.

On-policy training should help, and for structural reasons. The defect is that training states come from π^* and test states come from π_θ. Any method that generates from the current policy and scores those generations — rejection sampling into SFT, RLHF, RLVR — is closing exactly that gap. Rejection-sampling fine-tuning is, read this way, a poor man's DAgger where the "expert label" is a verifier's yes.

Anatomy of an SFT example

A conversation is a structured object: a list of turns, each with a role and a content string. A transformer consumes a flat sequence of integers. The chat template is the function between them, and it is one of the most consequential and least examined pieces of the stack.

Roles

Four roles cover essentially all modern practice:

The system role is doing more work than it looks like. It is what makes the same weights behave as a terse code reviewer in one deployment and a patient tutor in another, and the reason it works at all is that SFT data included examples where the system content varied and the assistant behaviour covaried with it. A model trained with a constant system prompt learns to ignore the system prompt, because it carried no information.

The tool role is where the conversation stops being a dialogue and becomes a trajectory with an environment in it. The assistant emits a structured call, execution happens outside the model, and the result comes back as a new turn the model did not write. From the loss's point of view a tool result is context — never a prediction target — for a reason that becomes obvious once you say it out loud: training the model to predict the output of a search API is training it to hallucinate search results.

Special tokens and why they must be special

Role boundaries have to be unforgeable. If the boundary between user and assistant were just the literal text "Assistant:", a user could type that string and the model would have no way to distinguish an instruction from a role switch. So role markers are added to the vocabulary as single tokens that the tokenizer never produces from user text:

<|begin_of_text|>                    BOS — one per sequence
<|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
Explain SGD in two sentences.<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>
SGD estimates the gradient ...<|eot_id|>   <-- generation stops here

Three details in that block carry real weight.

The turn-end token is the stop condition. Emitting <|eot_id|> is a learned behaviour, and it is the single most important thing SFT teaches that pretraining cannot: a base model has no concept of "done." If the turn-end token is under-represented in training, or if generation is run with a template that does not match the one used in training, the model will run past the end of its answer and start writing the user's next message. Every practitioner has seen this exact failure; it is almost always a template mismatch.

The generation prompt is part of the contract. At inference the template must emit the assistant header and then stop, so the model's first predicted token is the first token of the reply. Off-by-one errors here — a trailing newline present in training and absent at serving, or vice versa — put the model one token off-distribution at the most sensitive position in the whole sequence.

BOS is emitted exactly once, by the template. A recurring bug is a tokenizer configured to prepend BOS combined with a template that already includes it, producing a doubled BOS that appears in no training example.

Multi-turn examples are not one example

A conversation with three assistant turns contains three supervised decisions, each conditioned on a longer prefix. There is a choice about how to use it, and the choice matters:

The first is standard and the arithmetic is straightforward: with causal masking, scoring every assistant token in one pass gives exactly the same per-token gradients as the exploded version, because token t in turn 3 already conditions on all of turns 1 and 2. The exploded version is not more correct; it is the same computation done N times.

Packing, and the contamination it invites

SFT examples are short — often a few hundred tokens against a context window of tens of thousands — so padding to the context length wastes most of the compute. The standard fix is packing: concatenate examples until the buffer is full. The trap is that naive packing lets tokens in example B attend to example A through the causal mask, so the model is trained to condition an answer on an unrelated preceding conversation. The fix is a block-diagonal attention mask that resets at document boundaries, plus position ids that restart per example. Both are supported in modern training stacks and both are occasionally forgotten; the symptom is a model that inexplicably references content from a different conversation.

Which tokens receive loss?

This is the second of the two moves, and it is the one people get wrong. Once a conversation is a flat token sequence, you must decide which positions contribute to the loss. The general form introduces a binary mask m_t:

Setting m_t = 1 everywhere recovers pretraining. The usual choice for instruction tuning is assistant-only:

Note what does not change: masked tokens are still attended to. They are in the context, they shape every hidden state downstream, they just do not produce a gradient of their own at their own position. Masking is about the loss, not the attention.

Why assistant-only is the default

Three arguments, in decreasing order of how well they hold up.

You do not want to model the user. Scoring user tokens trains the model to generate plausible user turns. At best that is capacity spent on a distribution you will never sample from. At worst, at inference the model — having been trained that a user turn follows an assistant turn — writes one, and you get the classic runaway where the model plays both parts.

Prompt tokens are often boilerplate. A fixed system prompt repeated across a million examples, scored every time, is a million gradient steps toward memorising a string you will always provide anyway.

It matches what you are evaluating. The quantity you care about is p(response | prompt), and the assistant-only loss is exactly the negative log of that quantity. Full-sequence loss optimises p(prompt, response), which includes a term for how likely the prompt itself was — irrelevant to the task.

Except that it is not always right

The third argument is clean and the empirical picture is messier. Shi et al. found that including the loss over instructions — they call it instruction modelling — improved results across a broad benchmark suite in many settings, with two identifiable conditions governing when it helps: datasets with long instructions and short responses, and datasets with few training examples. Their explanation is regularisation. If responses are short and examples are few, an assistant-only loss has very few targets and is easy to overfit; adding the instruction tokens back gives the model more to predict and less room to memorise.

Which lines up with a way of thinking about the mask that I find more useful than the "don't model the user" heuristic:

Two further wrinkles that are easy to get wrong in practice:

The role headers themselves. Even under instruction modelling you generally still mask the special role tokens. The model should learn to emit <|eot_id|> at the end of its own turn — that is a real behaviour — but training it to emit the user header is training it to hand itself the microphone.

The normalisation. Dividing by the number of scored tokens per example and then averaging over the batch gives every example equal weight regardless of length. Summing over the batch and dividing by total scored tokens gives every token equal weight, so long examples dominate. These are different objectives and they produce measurably different models — the first pushes toward whatever short examples do, the second toward whatever long ones do. This interacts badly with packing, where "per example" becomes ambiguous, and it is a common source of results that fail to reproduce across training frameworks.

What does SFT actually teach?

"Fine-tuning makes the model better at X" hides four different mechanisms that behave differently, cost differently, and fail differently. Separating them is the most useful conceptual move in this whole article.

The evidence for elicitation is strong

LIMA is the sharpest data point. Zhou et al. fine-tuned a 65B base model on a thousand carefully curated prompt–response pairs — no RLHF, no preference model — and found it competitive in human preference comparisons against models trained on orders of magnitude more instruction data. They named the underlying claim the Superficial Alignment Hypothesis: knowledge and capabilities are learned almost entirely during pretraining, and alignment teaches the model which sub-distribution of formats to use when talking to a user. A striking ablation: coherent multi-turn behaviour emerged from adding a few dozen dialogue examples, despite the base set containing none.

Reinforcing evidence comes from the other direction. Work on in-context alignment showed base models can be pushed a long way toward assistant behaviour with a carefully constructed prompt and a handful of in-context examples — no weight updates at all. If a prompt can do a large share of what fine-tuning does, then what fine-tuning was doing was substantially selection among behaviours already available, not construction of new ones.

The evidence against knowledge acquisition is stronger still

Gekhman et al. ran the controlled version of this experiment. They partitioned closed-book QA fine-tuning examples by whether the model already knew the answer, varied the proportion that introduced genuinely new facts, and watched the training dynamics. Two findings, both mechanistically important:

First, examples carrying new knowledge are fitted substantially more slowly than examples consistent with what the model already knows. The model resists them. Whatever gradient descent is doing on a fact that contradicts nothing but is simply absent, it is not the fast interpolation it does on the rest of the data.

Second, and this is the part that should change practice: as those unknown examples are eventually fitted — late in training, as a form of overfitting — the model's tendency to hallucinate rises roughly linearly with how many of them it has absorbed. The best development-set performance occurs at the point where most of the known examples have been fitted and few of the unknown ones have. Early stopping is not just a regularisation nicety here; it is the mechanism that keeps the model from learning the meta-behaviour "confidently produce a fact-shaped string I have no grounds for."

But do not flatten this into "SFT only changes style"

The superficial-alignment framing is the most over-quoted result in post-training and it is worth being precise about what it does and does not support.

The LIMA evaluation was single-turn human preference on 300 prompts, with ties folded into "equivalent or preferred." That is a real measurement of one thing — whether a response reads as a good assistant answer — and it is close to silent on robustness, calibration, adversarial safety, long-horizon coherence, and precise constraint following. The paper's own analysis notes weakness on adversarially framed prompts. A model can be stylistically indistinguishable from a strong assistant on curated prompts and materially worse everywhere the curated prompts do not reach; that is exactly what §3 predicts.

There is also a clear counterexample class. Long chain-of-thought fine-tuning changes something that is hard to call style: models trained on extended reasoning traces produce backtracking, self-verification and enumeration behaviours at rates that differ qualitatively from the base model's, and the resulting accuracy gains on held-out competition mathematics are not explicable as formatting. Whether this is best described as acquiring a capability or as eliciting a latent one — sharpening the distribution around reasoning modes that pretraining already contained — is genuinely unresolved, and the sharpening-versus-expansion question recurs throughout the RL literature too. I do not think the field has settled it, and I would be suspicious of any article that tells you it has.

The practical consequence for domain fine-tuning

This has a direct and frequently ignored implication. The most common enterprise instinct — "we will fine-tune the model on our internal documents so it knows our product" — is asking SFT to do the thing it is worst at, and the failure mode is not a model that fails to learn. It is a model that learns the style of confidently answering questions about your product while having absorbed the underlying facts poorly and unevenly, which is a worse outcome than not fine-tuning, because it is harder to detect. Retrieval puts the facts in the context window, where the model's demonstrated strength — using information present in the prompt — actually applies. If fine-tuning on domain knowledge is genuinely required, the Gekhman result argues for filtering toward facts the model already partly knows, and stopping early.

The instruction-data taxonomy

"Instruction data" is not a category, it is a dozen categories with different construction costs, different verification stories, and different interactions with each other. Here is the working taxonomy, ordered roughly by how hard the data is to produce well:

The verifiability column is the one to read twice. It splits the taxonomy in half, and the split is what determines everything downstream: categories with a checker can be scaled synthetically with confidence and later handed to RL with verifiable rewards; categories without one depend on human judgement or a teacher model's taste, cannot be scaled the same way, and are where preference optimisation earns its place. This is the same split that structures the rest of the post-training pipeline.

The mixture is the actual design object

What you train on is not a dataset, it is a weighted combination:

Two things make this harder than it looks. Sources differ wildly in size, so proportional mixing lets one large source swamp everything — FLAN handled this by capping the number of examples drawn per dataset, and the capped-proportional scheme has been standard ever since. And sources interact: adding mathematics can move safety behaviour, adding refusal data can raise over-refusal on benign prompts, and the interaction sign is not predictable from the sources alone.

Tülu 3 is the most useful public artefact here because it publishes the mixture rather than describing it. Its SFT mix is roughly 940k examples across nineteen sources spanning general chat, knowledge recall, mathematics, coding, safety and non-compliance, multilingual data, and precise instruction following — with the mixture arrived at by building skill-specific mixes first, keeping what won on each skill in isolation, and then combining and re-ablating.

The mixture question is deep enough that it has its own article — negative transfer, catastrophic forgetting, replay ratios, mixture laws and merging are all mixture-level phenomena rather than SFT-level ones. Rather than duplicate it here, see One Model, Many Skills, which takes the weights λ_i as its subject.

How the data gets built

Every instruction dataset is produced by some combination of seven mechanisms. They differ in cost per example, in ceiling, and in what kind of error they introduce — and the last property is the one that decides which you can safely scale.

1. Human-written

An annotator writes the prompt, the response, or both. Highest quality per example, lowest throughput, and the only mechanism that can introduce genuinely novel behaviour rather than recombining existing behaviour. InstructGPT was built this way; so was the human-authored portion of LIMA and open efforts like No Robots and OpenAssistant. Expensive enough that it is now used surgically — for seeding, for categories nothing else can produce, and for the evaluation sets that keep everything else honest.

2. Benchmark conversion

Take an existing supervised NLP dataset and verbalise it as an instruction. This is what FLAN did: over sixty datasets, each rendered through multiple natural-language instruction templates, which turned a 137B base model into something that beat few-shot GPT-3 on a range of held-out task types. The FLAN ablations are the origin of the field's core empirical claim — that task diversity is what drives generalisation to unseen task types, more than any single dataset's quality. Cheap, and the labels are as good as the original dataset's.

The weakness is register. Converted benchmarks produce terse, benchmark-shaped answers, and a model trained only on them behaves like a leaderboard rather than an assistant. The Flan Collection's later work on mixing zero-shot and few-shot templates and rebalancing the mixture is largely about mitigating this.

3. Teacher-generated and distillation

Prompt a stronger model; keep its outputs. This is the mechanism behind most open instruction datasets from 2023 onward. It is cheap, fast, and can be steered by prompting the teacher for a particular style. Two structural costs: the student inherits the teacher's errors and its confidence in them, and the student's ceiling is bounded by the teacher on any behaviour it cannot verify independently. It also raises licensing questions that are outside the scope of a technical article but not outside the scope of your legal team.

4. Self-Instruct and evolution

Bootstrap instruction data from the model being trained. Self-Instruct seeds with a small set of human-written tasks, prompts the model to generate new instructions in the same spirit, filters for validity and near-duplicate overlap against everything generated so far, then generates inputs and outputs. Applied to vanilla GPT-3, it closed most of the gap to InstructGPT-001 with almost no annotation. The critical component is the diversity filter — without it, generation collapses onto a narrow region of instruction space almost immediately.

Evol-Instruct extends this by rewriting existing instructions into harder ones along explicit axes — add constraints, deepen, concretise, increase reasoning steps — which addresses the complementary failure: a bootstrapped set that is diverse but uniformly easy.

5. Rejection sampling

Generate k candidates from the current model, keep the ones that pass a check, train on those. This is the workhorse of modern pipelines and it deserves its own line of reasoning.

It is on-policy in exactly the sense §3 said matters: the states being labelled are the states the model actually visits, which is the defect behaviour cloning has. Yuan et al. formalised it for mathematics as rejection-sampling fine-tuning and found something that shaped subsequent practice — the gain depends on the number of distinct reasoning paths among the retained samples, not the raw count of retained samples. Ten copies of the same derivation teach almost nothing; ten different valid derivations of the same answer teach the model that multiple routes exist. Combining rejection samples across several models pushed LLaMA-7B on GSM8K well past what supervised fine-tuning on the original data achieved.

6. Execution and verification

Where a checker exists, use it as a filter and the pipeline changes character entirely: unit tests for code, answer matching for mathematics, parsers for structured output, constraint checkers for precise instruction following. Tülu 3's persona-driven synthesis is a clean example — seed with a small set of hand-written verifiable constraints, generate many more against a persona set, and keep only those where the response provably satisfies the constraint.

7. Filtering and selection

The last stage, and the one with the best return per unit of effort. AlpaGasus found that scoring an existing instruction set with a model judge and keeping a small high-quality fraction produced a better model than training on all of it. Deita generalised this into explicit measurement along three axes — complexity, quality, diversity — and matched much larger baselines with about 6k selected examples. LIMA arrived at the same place from the manual direction.

The converging finding across all of these is worth stating plainly: for instruction tuning, adding more mediocre data frequently makes the model worse, not merely no better. That is unusual — it is not how pretraining behaves — and it follows directly from the mode-covering property of forward KL in §2. Every mediocre demonstration is a mode the model is being pressured to cover.

Decontamination is not optional and it is not easy

Synthetic pipelines will happily regenerate your evaluation set. A teacher model asked for "a challenging grade-school word problem" produces something close to GSM8K because GSM8K is in its training data. Tülu 3's contribution here is as much the decontamination tooling and the discipline of holding out genuinely unseen evaluations as it is the mixture. If your pipeline includes a teacher model and you have not run n-gram and embedding-level overlap checks against every benchmark you report, your numbers are decorative.

Reasoning SFT

The most consequential single decision in a modern SFT set is whether the target is

and the reason it matters is architectural, not stylistic.

Why an intermediate rationale changes what is computable

A transformer computing p(a | q) in a single forward pass has a fixed compute budget: L layers of processing, one pass, no matter how hard the question is. That is a serial-depth bound, and it is not negotiable by making the model wider or training it longer.

Now write the answer as a marginal over intermediate states:

When the model emits r token by token before a, each rationale token gets its own full forward pass, and every subsequent token can attend to it. A rationale of length T converts a depth-L computation into something closer to depth L × T with an external write-once memory. This is why chain-of-thought helps on problems requiring sequential composition and helps almost not at all on problems answerable by lookup: it buys serial depth, and only some problems are bottlenecked on serial depth.

Training on (q, a) pairs alone therefore does something specifically bad on hard problems. It demands that the model produce the marginal in one pass. Where the question genuinely requires composition, the only representable strategy is guess a plausible-looking answer, and gradient descent will happily fit that, because on the training set the guesses were labelled correct. You have trained a model to assert.

Where the rationales come from

Chain-of-thought distillation. Prompt a strong model to think step by step, keep the trace. Cheapest, and inherits the teacher's reasoning style wholesale — including its characteristic errors and its length.

Bootstrapped rationales. STaR's loop: sample a rationale, keep it if the answer is right, retrain, repeat; for problems the model cannot solve, provide the answer as a hint and let it rationalise backward. This is expectation-maximisation with the rationale as the latent variable, done approximately, and it works because correctness of the answer is a cheap noisy signal about correctness of the path.

Long chain-of-thought traces. The traces produced by reasoning-trained models look different in kind: they are much longer, and they contain explicit backtracking, hypothesis abandonment, self-verification and restarts. Fine-tuning a non-reasoning model on a small set of these transfers a surprising amount. LIMO and s1 both landed on roughly a thousand carefully chosen examples with long traces — s1K selected on difficulty, diversity and quality — being enough to move a strong instruct model substantially on competition mathematics. This is the LIMA result again, one level up: if the base model already contains the components, a small number of well-chosen demonstrations can reorganise how they are deployed.

The outcome-correct, rationale-wrong problem

Every scalable rationale pipeline filters on the final answer, because the final answer is the only thing cheaply checkable. That admits false positives: traces that reach the right answer through invalid reasoning — an arithmetic slip cancelled by a second slip, an unjustified leap that happens to land, a "therefore" doing work no preceding line supports.

These are not rare. Systematic study across open models, difficulty levels and decoding strategies finds false-positive solutions persist across all of them, that sampling-based inference-time scaling does not remove them, and that pass@N is especially inflated by them — implying a lower real ceiling than automatic evaluation suggests. And in a self-improvement loop the effect is worse than a constant tax, because you are training on them, so you are actively reinforcing invalid inference patterns that happen to be answer-preserving.

Short traces or long ones?

There is a real trade-off and it is not resolved. Long traces raise accuracy on hard problems and are what the current reasoning models are built on. They also cost inference tokens linearly, encourage the model to spend a thousand tokens on a question that needed ten, and train a length habit that transfers to tasks where it is pure overhead. Concise traces are cheap and often sufficient, but they cap what the model can do on problems that genuinely need the depth.

The pragmatic answer in current recipes is to train on both and make the length controllable — through the data mixture, through explicit budget signals in the prompt, or through a separate thinking mode. Whether length control is best learned in SFT or imposed at inference is an open engineering question, and it is the subject of the test-time-scaling literature rather than this article.

What reasoning SFT is unambiguously good at is initialisation. A model that has been shown what a long, structured, self-correcting trace looks like starts reasoning RL from a policy that already emits the right shape of output, which is why the DeepSeek-R1 recipe uses a cold-start SFT stage before reasoning RL, and uses rejection-sampling SFT again after it. SFT and RL alternate in modern pipelines; they are not sequential phases you pass through once.

How much SFT?

SFT is a small perturbation to a large object. The pretrained model represents something on the order of 10²⁴ FLOPs of accumulated computation; a typical SFT run is four to six orders of magnitude smaller. Every hyperparameter choice should be read through that asymmetry: the question is never "how do I fit this data" but "how do I move the model as little as possible while still moving it enough."

The knobs, and what each one actually controls

The epoch range is worth dwelling on because it looks contradictory. Large mixtures get two to four passes; tiny curated sets like LIMA get fifteen. Both are right, and the resolution is that what matters is roughly total gradient steps against the size of the model, not passes over the data. A thousand examples at fifteen epochs is fifteen thousand example-visits, which is still a rounding error next to a million-example mix at three epochs. The genuinely dangerous regime is a large mixture trained for many epochs, which is where memorisation of specific responses starts showing up in generations.

Overfitting here does not look like overfitting

The training loss going down while validation loss goes up is the textbook picture, and it is not the useful signal for SFT. Three things happen before validation loss turns, and all of them matter more:

Verbatim regurgitation. The model starts reproducing specific training responses for prompts that merely resemble the training prompt. Diagnostic: sample at temperature 0 on a held-out set and check for near-duplicate matches against training responses.

Diversity collapse. Sample the same prompt twenty times and get twenty near-identical answers. This is the forward-KL objective doing exactly what it was told — concentrating on demonstrated modes — carried too far, and it is measurable long before any loss curve moves. It also badly damages downstream RL, which needs sampling diversity to have anything to learn from.

Hallucination onset. The Gekhman dynamics from §6. Unknown examples are absorbed late; the model's willingness to assert unsupported facts climbs with them. This is the most damaging and the least visible, because it does not show up on any benchmark that only measures the tasks you trained on.

Length drift is a real and under-monitored failure

Response length has a habit of climbing during post-training, and it climbs for banal reasons: teacher models are verbose, human annotators equate length with effort, and evaluation judges — human and model — measurably prefer longer answers. If your data is verbose, your model will be. If your judge rewards verbosity and you select data by that judge, you have built a length-amplifying loop.

Track median response length per category, per checkpoint, alongside your quality metrics. It costs nothing and it catches a drift that quality metrics will actively hide from you, since the judge causing the drift is the one grading it.

Catastrophic forgetting, in its real proportions

A model fine-tuned hard on one domain gets worse at others. That much is well established, and it worsens with learning rate, with epochs, and with the narrowness of the mixture. But the framing "the knowledge was destroyed" is usually wrong, and the distinction is practical:

Where the loss is real, the mitigations are the ones that already appear in every part of this pipeline: keep the learning rate low, keep the mixture broad, replay a fraction of general instruction data alongside the specialised data, and stop early. Parameter-efficient methods change the trade-off — constraining the update to a low-rank subspace both learns less and forgets less — rather than eliminating it. The mixture-level treatment of all of this is in One Model, Many Skills.

Where SFT sits in the pipeline

The tidy story is linear: base model → supervised fine-tuning → preference optimization → reinforcement learning → specialized capabilities. The animated map below adds the important part that shorthand omits: later stages generate data for earlier ones.

Tülu 3 is the clearest open instantiation: curated prompts, then SFT, then DPO, then RLVR, with a published evaluation scheme separating development from genuinely unseen benchmarks. It is worth reading precisely because it documents what did not work as well as what did.

But the linear picture is at best a first approximation, and the honest version is a loop. R1-style recipes run cold-start SFT, then reasoning RL, then another round of rejection-sampling SFT on the RL model's own best outputs, then a final RL stage. Each of these is SFT; each is on data the previous stage generated. And the reason the loop exists is exactly §3: SFT's defect is that it trains on the expert's state distribution, RL's contribution is that it samples from the model's own, and rejection-sampling SFT is the cheap way to convert the latter back into the former.

What is still unsettled

Six things I do not think the field has resolved, stated as questions rather than as positions.

1. Is long-CoT fine-tuning elicitation or acquisition?

A thousand long reasoning traces move a strong instruct model substantially on competition mathematics. Two readings. Either the base model already contained backtracking and verification behaviours and the traces taught it when to deploy them — elicitation, the LIMA story one level up. Or the traces installed a genuinely new procedure. The evidence is compatible with both, the experiments that would separate them are expensive, and the same ambiguity — sharpening an existing distribution versus expanding its support — recurs unresolved in the reasoning-RL literature. I would not bet on either reading.

2. "SFT memorises, RL generalises" is a slogan doing more work than the evidence supports

Chu et al. is real work with a real finding: in two rule-based environments, RL with outcome rewards transferred to unseen rule and visual variants while SFT largely did not — and the same paper found SFT remained necessary to stabilise output format before RL could work at all. That is a specific result in a specific setting. It has been widely flattened into a general law about the two methods, and the flattening is not warranted: most SFT-versus-RL comparisons in the literature differ simultaneously in the data, the amount of compute, whether training is on-policy, and how hyperparameters were tuned. Some of the gap is very likely the on-policy/off-policy distinction from §3 rather than anything intrinsic to maximum likelihood. Careful matched comparisons are still scarce.

3. Nobody can predict how much data a given capability needs

A thousand examples is enough for assistant behaviour. Tülu 3 uses close to a million. Both are defensible; there is no theory that tells you in advance which regime you are in for a capability you have not tried yet, and the empirical answer differs by orders of magnitude across capabilities. Data-selection methods help you spend a budget well, but they do not tell you what the budget should be.

4. The masking and normalisation choices have no principled answer

Assistant-only is the default, instruction modelling wins in identifiable regimes, and the per-example versus per-token normalisation choice materially changes the model with no theory saying which is correct for a heterogeneous mixture. These are consequential decisions currently made by convention and by whichever training framework you happened to adopt.

5. Filtering rationales on outcomes is known to be wrong and still universal

False positives get through and get trained on. Step-level verification would fix it and costs far more per example than the outcome check. Process reward models are one route, model-based trace critique another, and neither is cheap enough to have displaced answer matching at scale. The size of the tax this imposes on current reasoning models is not well characterised.

6. The recursion has not been evaluated over enough generations

Modern SFT data is largely generated by models, filtered by models, and scored by models. Each of those introduces a bias toward the generating distribution, and each generation of models is trained on data shaped by the previous. Over one or two hops this appears to be fine and is clearly useful. What it does over five, with the filters themselves drifting, is not something anyone has measured, and the sampling that would answer it takes years of calendar time.

References

Grouped by topic. Every entry links to its primary source.