How LLMs Are Pretrained

A one-line objective

A pretraining run has exactly one loss function, and it fits on one line. Given a token sequence x, the model is trained to minimise the negative log-likelihood of each token given everything before it:

That is the whole thing. There is no auxiliary loss, no contrastive term, no reconstruction head, no reward. A modern frontier pretraining run is this expression, evaluated a few hundred quadrillion times.

Which raises an obvious question. If everybody is minimising the same objective with roughly the same architecture and roughly the same optimiser, why are the resulting models so different?

The answer is hiding in a subscript. The loss is defined over a sequence x, and x is drawn from a distribution D. Write the loss the way a statistician would and the missing variable appears:

The objective is simple. The distribution being optimised is not. Nobody hands you D. There is no natural distribution over “text.” Every pretraining team constructs one, out of crawls and repositories and books and synthetic generations, through a pipeline of a dozen irreversible decisions. That constructed distribution is the actual product of a pretraining effort. The weights are just its fixed point.

This article is about how D gets built. The previous article answered how much — how parameters, tokens and compute trade against each other. This one answers which and how: which text, filtered how, deduplicated how, mixed in what proportion, tokenized with what vocabulary, packed into what sequences, ordered in what curriculum — and how you find out, before you have burned ten million GPU-hours, whether any of it was a good idea.

Before anything else, it is worth being concrete about what one training example actually is, and where the loss comes from. The figure below walks a single sequence through the objective: the tokens, the predicted distribution at each position, the log-probability picked out by the true next token, and the sum.

Where the data came from

Pretraining corpora have a history, and the history explains most of the present. Each generation was a response to a specific failure of the one before it.

Why the field moved this way. Early models could afford hand-curated corpora because their token budgets were small. Scaling exhausted those sources, forcing a move to the web; web noise then made filtering and deduplication central. Once cleaning became competent, composition—not raw volume—became the lever. Today aggressive filtering collides with finite unique data, so OCR recovery, code and mathematics, multilingual sources, and carefully generated synthetic text fill the gap.

The durable lesson across these eras is that a dataset is a sequence of decisions, not a pile of documents. Crawling determines coverage; extraction determines what text survives; filtering establishes a quality prior; deduplication controls wasted compute and memorization; and sampling weights decide which capabilities receive gradient signal. Improvements attributed to a named corpus usually came from changing several of those axes together.

The modern shift is from rules justified by intuition to decisions justified by training ablations. Open pipelines made it possible to vary one filter at a time; standardized data benchmarks made those results comparable; and frontier-scale recipes exposed the next constraint: the cleanest slice is not large enough to repeat forever. The practical compromise is a broad real-text pool plus smaller, intentionally weighted streams of code, mathematics, OCR-recovered documents, multilingual text, and synthetic teaching material.

Raw web is not training data

A Common Crawl snapshot is delivered as WARC files: raw HTTP responses, headers and all. Turning that into something a language model should see requires a pipeline. Here is the canonical ordering, which is remarkably consistent across FineWeb, DCLM, Dolma, RefinedWeb and Nemotron-CC even though their individual choices differ:

Two structural facts about this list matter more than any individual step.

Every step is lossy and irreversible. Once a document is dropped, no downstream stage can recover it. The corpus is a chain of filters, and the model can only learn from the intersection.

Order changes the outcome. Deduplicating before quality filtering and deduplicating after produce different corpora, because dedup decides which copy of a near-duplicate survives and the filters then judge that particular copy. Similarly, running language ID before or after boilerplate removal changes which pages are classified as English, because navigation chrome is often English on a non-English page.

Step by step, and what each one costs

HTML extraction. Common Crawl ships both WARC (raw) and WET (pre-extracted plain text) files. Using WET is free; using WARC and running your own extractor is expensive and better. FineWeb used Trafilatura on raw WARC; DCLM used Resiliparse and measured the difference directly. This is the single most surprising line item in any pipeline budget: the choice of HTML-to-text library measurably changes downstream MMLU. Boilerplate removal is not a preprocessing detail, it is a data decision.

Language identification. Usually a fastText classifier with a confidence threshold — FineWeb keeps English above 0.65. The threshold is a lever with a real trade-off: raise it and you lose code-switched text, technical documents with heavy notation, and short pages; lower it and you admit machine translation and gibberish. For multilingual corpora, the same classifier decides how many tokens each of 100+ languages receives, which means a single fastText model is silently setting your multilingual mixture.

Quality heuristics. The Gopher and C4 rules, largely intact a half-decade later: reject documents with too few words, too many repeated lines, too high a repeated n-gram fraction, too low a stopword ratio, too many symbols relative to letters, too many bullet points, lines that do not end in terminal punctuation. Every one of these is a hand-written guess about what “real prose” looks like, and each one has a failure mode — the terminal-punctuation rule, for instance, deletes most tables, most poetry, and much code.

Safety filters. URL blocklists (the UT1 adult-content list is near-universal), toxicity classifiers, PII detection and redaction. These have a documented and awkward property: toxicity classifiers disproportionately remove text written in African-American English and text about LGBTQ+ topics, which means safety filtering is also a demographic filter on whose language survives into the corpus.

Deduplication. Large enough to deserve its own section; see below.

Domain classification. Assigning each surviving document to a bucket — web, code, maths, science, encyclopedic, conversational — so that the mixture step has something to weight. Usually a classifier, sometimes a URL heuristic, sometimes both. Errors here propagate directly into mixture weights, and there is no clean ground truth: is a Stack Overflow answer “code” or “conversation”?

Contamination removal. Stripping documents that overlap with evaluation benchmarks, usually by n-gram matching against test sets. This is the step everyone agrees is necessary and nobody performs identically, which is why cross-lab benchmark comparisons are shakier than they look.

Deduplication

The web is enormously redundant. The same article appears on the original site, three aggregators, a scraper farm, an archive mirror and a translation-and-back-translation loop. Licence boilerplate appears on millions of pages. Wikipedia is mirrored so widely that a naive crawl contains hundreds of copies of every article.

Deduplication is the highest-leverage step in the entire pipeline, for three separate reasons that people often collapse into one.

Reason one: efficiency

Duplicated text wastes gradient steps. If a document appears fifty times, the model spends fifty times the compute on it, without fifty times the information. Removing duplicates lets a fixed compute budget cover more distinct text. This is the boring reason and the smallest one.

Reason two: memorization and privacy

This is the reason that matters. Memorization is not uniform across the corpus — it is strongly driven by duplication count. A sequence seen once is unlikely to be reproduced verbatim; a sequence seen a thousand times is very likely to be. Carlini and colleagues quantified this directly: extractable memorization scales log-linearly with the number of duplicates in the training data, and also with model size and with context length.

Which means deduplication is a privacy control, not just an efficiency measure. A phone number that appears in one leaked pastebin is probably safe; the same number in a document mirrored across ten thousand scraper sites is at real risk of being emitted verbatim. Lee and colleagues showed that models trained on deduplicated corpora emit memorized training text roughly an order of magnitude less often, while performing no worse.

Reason three: evaluation validity

Duplicates cross the train/test boundary. If a document appears in both your training corpus and your held-out validation set, your validation loss is measuring memorization, not generalisation. Standard corpora were found to have substantial train/test overlap once anybody bothered to check. Deduplication is what makes your held-out set actually held out.

The methods, from cheapest to most thorough

Exact document dedup. Hash the full document, drop repeats. Trivial to implement, cheap to run at any scale with a Bloom filter, and catches almost nothing — a single differing timestamp or advertisement defeats it. It is the free baseline everybody runs first.

Near-duplicate document dedup via MinHash + LSH. The workhorse. The idea is to make Jaccard similarity computable without comparing every pair.

Substring / suffix-array dedup. Document-level methods miss the case where two otherwise different documents share a long identical passage — a quoted licence, a repeated news wire paragraph, a boilerplate footer. Suffix-array methods find every substring of length ≥ k (typically 50 tokens) that appears more than once in the corpus and remove the repeats. This is exact, catches what MinHash cannot, and is the more expensive of the two. It also has a side effect people underrate: it fragments documents, leaving stitched-together text whose sentences no longer follow each other.

Semantic / embedding dedup. Cluster document embeddings and thin dense regions. Catches paraphrase and translation duplicates that share no n-grams at all. Expensive, less standard, increasingly common in image and multimodal pipelines.

The scope question: global or per-source?

This is where intuition fails most reliably. Deduplicating globally — across the entire corpus at once — feels obviously correct. FineWeb tried it and found the opposite. Independently deduplicating each Common Crawl snapshot outperformed global deduplication across all snapshots.

The explanation is a selection effect, and it is worth sitting with. A document that appears in many crawl snapshots is a document that stayed online for years across many domains. That is a quality signal. Global dedup keeps exactly one copy of it and deletes the rest, which flattens that signal to nothing; worse, in the snapshots where it was removed, what remains is disproportionately the transient junk that only appeared once. Aggressive global dedup can leave you with a corpus enriched in ephemera. The upsampling that duplication provides was doing useful work.

The FineWeb team also measured the alternatives directly: URL-level global dedup removed 71.5% of tokens, leaving 5.6T of the 20T that per-snapshot MinHash had produced, and did not help. Dolma 3, by contrast, invested in trillion-token-scale global dedup tooling and did use it — alongside a separate quality-aware upsampling step that puts the duplication signal back in deliberately rather than accidentally. That is the reconciliation: global dedup is fine if you re-introduce frequency information explicitly instead of relying on raw copy counts to carry it.

What “quality” means

Every pipeline has a step called “quality filtering,” and the word is doing something dishonest. There is no property of a document called quality. There is only usefulness for training a model that will be evaluated in a particular way, and that is a much stranger thing.

Consider what the objective actually rewards. Pretraining maximises likelihood. A corpus of pure repetitive boilerplate would give you spectacular validation loss and a useless model. A corpus of maximally surprising text — random character strings — would give terrible loss and an equally useless model. Neither extreme is what we want, and the objective by itself cannot tell them apart. So “quality” is smuggled in from outside, as a proxy.

The proxies people actually use

Heuristic quality. The Gopher/C4 rule stack. Cheap, interpretable, easy to audit, and blunt. Best understood as removing things that are definitely not prose rather than selecting things that are definitely good.

Perplexity under a reference model. Train a small model (classically a KenLM 5-gram model on Wikipedia) and keep documents it finds neither too surprising nor too predictable. CCNet popularised this. The failure is circular: you are selecting for text that resembles your reference corpus, so your reference corpus's blind spots become the model's.

Classifier scores. Train a binary classifier, keep the top k%. The interesting question becomes what you use for positives. DCLM used a fastText classifier with instruction-formatted data and ELI5 answers as positives, RefinedWeb as negatives, keeping the top ~10%. That single choice was worth more than 6 MMLU points over the previous best open corpus — and swapping the positive set moved MMLU across a 35%–44% range. The classifier is not detecting quality; it is detecting resemblance to whatever you chose to call good.

Educational value. FineWeb-Edu's approach: have Llama-3-70B rate 500k web pages on a 0–5 educational scale, distil those judgements into a small BERT-style regressor, run it over the whole corpus. Keeping only pages scoring 3+ removed 92% of FineWeb and left 1.3T tokens. In an ablation at 1.8B parameters and 350B tokens, MMLU rose from 33% to 37% and ARC from 46% to 57%. Classifying all 15T tokens cost 6,000 H100-hours — a rounding error against the training run it improves.

Source reputation. Whitelist arXiv, PubMed, Wikipedia, Stack Exchange, curated book collections. Reliable and small. This is the “let humans do the filtering” heuristic again, now with institutions instead of Reddit.

Target-task matching. The newest and most honest framing: stop pretending there is a task-independent notion of quality, and select documents by similarity to the distribution of tasks you actually care about. This makes the circularity explicit rather than hiding it, at the cost of building a model that is good at the things you already thought to measure.

Choosing what goes in

Data selection and data mixing are usually discussed together and they are not the same problem. Selection is per-document: does this page belong in the pool at all? Mixing is per-domain: given the pool, how often should the sampler draw from each bucket? Different mathematics, different failure modes, different tooling. This section is selection; the next is mixing.

Four families, roughly in order of increasing sophistication and decreasing interpretability.

1. Rule-based rejection

Write predicates, drop what fails. Every pipeline starts here and none of them leave. The virtue is auditability: when a filter surprises you, you can read it. The vice is that rules encode a monolingual, prose-shaped idea of text, and quietly delete anything structured.

2. Model-based scoring

Train a scorer, threshold it. All the variants above — perplexity, fastText, educational regressors — are this. The decisive design question is not the model class but the definition of the positive set, because that set is the entire specification of what you want. Everything else is implementation.

There is a compute argument here that is easy to miss. A classifier pass over 15T tokens costs a few thousand GPU-hours. The training run it is filtering for costs hundreds of thousands. Spending 1% of your budget deciding what the other 99% will see is obviously correct, and until roughly 2023 almost nobody did it.

3. Target-aware selection

Rather than a generic quality score, score documents by how much they resemble a set of target tasks — either by embedding similarity to benchmark examples, or by importance resampling against a target distribution. The 2025 result that made this mainstream is straightforward: models improve when pretraining data matches target tasks, and the improvement is large enough that generic quality filtering looks like a weak special case (target = “text a helpful assistant would produce”).

The obvious objection is that this is contamination wearing a lab coat. It is not — you are selecting on distributional similarity to a task family, not on the test items themselves — but the boundary is genuinely narrow, and it is why decontamination has to be run after selection rather than before.

4. Influence and learnability signals

The most direct formulation: keep the documents whose inclusion most reduces loss on what you care about. Approximations include gradient-alignment methods, small-model influence estimates, and cheaper proxies such as attention-head influence. All of them are attempts to estimate the same intractable quantity — the derivative of final performance with respect to including one document — and all of them are expensive enough that they are used to label a subset and train a cheap classifier on the labels, rather than run over the full corpus.

The mixture problem

Suppose the pool has been built and partitioned into K domains:

The sampler draws domain i with probability wi, where w lies on the simplex (wi ≥ 0, ∑ wi = 1). The training distribution is the mixture

and we want the w that minimises whatever we actually care about — downstream loss, a benchmark aggregate, some weighted combination:

Stare at that for a moment, because it is a genuinely nasty optimisation problem. The objective requires training a model to convergence to evaluate a single point. Its dimension is the number of domains, typically 8–30. It is non-convex. There are no gradients with respect to w. And the function you can actually evaluate at small scale is not the function you want to minimise at large scale.

Nearly every method in this literature is a strategy for making this evaluable.

Strategy 0: a human writes the table

GPT-3, The Pile, Llama 1 and 2, and most production models through 2023. Someone sits down, decides that code should be 15% and Wikipedia should be upsampled 3×, runs the model, and adjusts next time. This works better than it has any right to, because the loss surface over w is quite flat near sensible mixtures. It is also unfalsifiable and does not transfer between corpora.

Strategy 1: proxy models and robust reweighting (DoReMi)

DoReMi's construction is elegant. Train a small reference model on some baseline mixture. Then train a second small proxy model with Group DRO, which maintains domain weights and continually shifts mass toward the domains where the proxy's excess loss over the reference is largest. Take the time-averaged weights the proxy converged to, and use them to train the real model.

The intuition: a domain where the proxy is much worse than the reference is a domain with headroom left; a domain where it already matches is one where extra tokens are wasted. The weights come out of a minimax game rather than a search.

Notably, the resulting weights are computed without reference to any downstream task. That is either the method's great virtue (no contamination, no target-task overfitting) or its central limitation (it optimises for balanced language modelling, which is not what you are evaluated on), depending on your view.

Strategy 2: fit a functional form (data mixing laws)

The Data Mixing Laws paper made the key observation: loss is a predictable function of mixture proportions, with a usable parametric form. For a target domain, per-domain losses combine roughly as exponentials in the weights:

Fit the parameters on a handful of small runs at sampled mixtures, then predict the loss of unseen mixtures and optimise analytically. And because you also have scaling laws in model size and training steps, you can nest them: fit the mixing law at small scale, extrapolate along the size and step axes, and predict which mixture will be best for the run you can only afford to do once.

Strategy 3: treat it as regression (RegMix)

RegMix strips away the parametric assumption. Train many tiny models on randomly sampled mixtures, fit a regression (gradient-boosted trees work fine) from w to validation loss, then search the simplex against the surrogate. It reported matching or beating DoReMi at roughly 10% of the compute, scaling up to 7B models on 100B tokens.

This is the surrogate-model pattern from classical hyperparameter optimisation, applied to the corpus. Once you see it that way, the whole later literature is legible: Bayesian optimisation variants, multi-fidelity variants, model-merging variants that avoid proxy training altogether.

Strategy 4: adapt the mixture during training

Nothing forces w to be constant. Aioli unified the offline methods under a single mixing-law parameterisation, then estimated its parameters online during the target run and adjusted proportions with exponentiated gradient descent — and showed that several earlier online methods were, in effect, setting those parameters badly. Bandit formulations and gradient-influence methods make related moves.

The practical objection to all of them is that dynamic mixtures interact badly with the rest of the stack: data loaders are built to stream fixed shard manifests, checkpoint–restart must reproduce the schedule exactly, and a mixture that changes with loss makes runs non-reproducible. Most production systems therefore use a small number of fixed stages instead of a continuously varying w — which is the curriculum question, §10.

The constraint everybody hits: token supply

All of the above treats w as freely choosable. It is not. If maths is 2% of your pool and the optimiser wants 8%, you cannot conjure six points of maths — you can only repeat what you have, which means the effective quality of those tokens decays (§11).

This is why the modern framing separates pool from mix. Dolma 3 has a 9.3T-token pool and draws a 5.9T-token mix from it. The mix is not a subset chosen by quality alone; it is the solution to a constrained problem: hit the target weights, subject to each domain's available supply, subject to a repetition budget per domain. Ai2 named the two halves of this explicitly — token-constrained mixing for the allocation, and quality-aware upsampling for deciding which documents within a domain get repeated when a domain must be repeated at all.

There is a second, subtler supply constraint. Olmo 3's authors note that they deliberately excluded structured data — question–answer pairs, chat-templated text — from pretraining even where it was available, because including it disproportionately moves evaluation results and makes data ablations harder to interpret. Some data is held out of pretraining not because it is bad but because it would contaminate your ability to measure. That is a strange and revealing constraint.

Tokenization

The tokenizer is the last irreversible decision before training, and it is the one most often made by copying someone else's file. It deserves better, because it determines the units the loss is even defined over.

BPE in one paragraph

Byte-pair encoding starts from a vocabulary of individual bytes and repeatedly merges the most frequent adjacent pair, recording each merge, until the vocabulary reaches a target size. At inference the merges are replayed greedily in the order they were learned. Byte-level BPE (GPT-2 onward) starts from all 256 bytes so that nothing is ever out-of-vocabulary, at the cost of representing rare characters as several tokens.

Two consequences follow immediately and matter enormously. First, the merge table is learned from a corpus, so the tokenizer inherits that corpus's distribution — a tokenizer trained on English web text will shred Telugu. Second, merges are greedy and deterministic, which means truncating a large vocabulary by merge rank produces exactly the tokenization you would have got by training at the smaller size. That is a useful practical fact: you can train one 200k vocabulary and derive every smaller ablation from it for free.

Fertility, and why it is a budget line

Fertility is tokens per word (or per byte, or per character — state which). It is the compression ratio of your tokenizer on a given text, and it converts directly into money.

Choosing a vocabulary size

Bigger vocabularies compress better (lower fertility, fewer tokens per document) and cost more in embedding parameters, output-projection FLOPs and logits memory. The trade-off is real and quantifiable, and the answer has drifted upward: GPT-2 used 50k, Llama 2 used 32k, Llama 3 moved to 128k, Olmo 3 uses ~100k, and recent large models have gone to 200k.

There is a systems consequence that gets discovered painfully. The logits tensor is batch × sequence × vocabulary, in fp32 for the cross-entropy. At 100k vocabulary, a batch of 8 sequences of 8192 tokens is 8 × 8192 × 100000 × 4 bytes ≈ 26 GB for one transient tensor. This is why fused and chunked cross-entropy implementations exist, and why vocabulary size is a memory decision as much as a modelling one. (The training-step article covers where that memory goes.)

Two recent directions are worth knowing. SuperBPE relaxes the whitespace pre-tokenization constraint partway through training, learning “superword” tokens that span spaces and cutting sequence length by around a third at 200k vocabularies. Byte Latent Transformer drops the fixed vocabulary entirely, grouping bytes into dynamically sized patches by local entropy — more bytes per patch where the text is predictable, fewer where it is not. And 2026 work on compute-optimal tokenization found something clarifying: when you measure the scaling relationship in bytes per parameter rather than tokens per parameter, an optimal compression rate appears, and it holds whether the compression comes from a BPE vocabulary or a learned latent grouping.

Numbers, code, and the things tokenizers break

Number handling is a design decision with visible downstream effects. If “2024” is one token and “2025” is two, arithmetic becomes needlessly positional. Most modern tokenizers split digits individually or in fixed groups of three for exactly this reason. Code needs indentation runs as single tokens or Python files explode in length. And any tokenizer trained predominantly on prose will handle LaTeX, chemical formulae and tabular data badly — which then shows up as poor performance on maths and science benchmarks that gets misdiagnosed as a data-mixture problem.

Sequence construction

The tokenizer emits one long stream per document. The model consumes fixed-length sequences. Getting from one to the other is the least glamorous step in this article and a reliable source of silent bugs.

Three options, one of which is correct

Pad. One document per sequence, pad to length. Wastes compute proportional to the padding — and web documents are short and heavy-tailed, so a 8192-token context filled with median-length web pages wastes most of itself. Nobody pretrains this way.

Concatenate and chunk. Join every document end to end with an EOS token between them, then slice the stream into equal-length blocks. Zero waste. This is what almost every pretraining run does. The cost is that documents get cut mid-sentence and every sequence contains fragments of several unrelated documents.

Pack with bin-packing. Fit whole documents into sequences with a first-fit-decreasing style heuristic, accepting a few percent padding to avoid splitting documents. Common in fine-tuning, less so in pretraining.

The cross-document attention question

Under concatenation, a naive causal mask lets token 5,000 (in document C) attend to token 12 (in document A). Should it?

The argument for allowing it: it is free, it acts as a mild regulariser, and separator tokens teach the model that an EOS means “forget what came before.” The argument against: it is training-time noise that never occurs at inference, and it corrupts long-range attention statistics precisely when you are trying to teach long context.

The empirical answer has firmed up. For general pretraining the effect is small. For long-context stages it is not — if you are training at 32k or 128k specifically to learn long-range dependency, filling those windows with a dozen unrelated documents and letting attention cross between them teaches the wrong thing. Modern stacks therefore use a block-diagonal document mask, built as a block mask so attention kernels can skip the zero blocks entirely rather than materialising them. And crucially, position IDs must reset at document boundaries too — otherwise the third document in a packed sequence sees rotary positions in the tens of thousands, and the model learns a spurious association between large position index and mid-document context. (The positional-encoding article has more on why that matters.)

Length curriculum

Attention is quadratic in sequence length, so training at the final context length from step one is wasteful. Nearly everyone therefore trains the bulk of the run at a short context and extends late. Qwen3 did its first 30T tokens at 4k and extended in a final stage. Olmo 3 dedicates a separate third stage of 50B tokens to long documents specifically. Llama 3 extended context in six stages.

The subtlety is that a long-context stage needs genuinely long documents, and those are rare. Concatenating short documents to fill a 128k window teaches nothing about long-range dependency — it teaches the model that distant context is irrelevant, which is the opposite lesson. This is why Olmo 3's long-context mix is described as drawing on longer documents specifically, and why PDFs (which are naturally long and coherent) are so valuable at this stage.

Curriculum and annealing

Ask the question directly: should every token have the same probability of appearing at every point in training?

For most of the field's history the answer was yes, by default rather than by argument — a fixed mixture, sampled i.i.d., start to finish. That default is now gone, and it was killed by an observation about learning rates rather than about data.

Why the end of training is special

Standard schedules decay the learning rate to near zero over the run. Near the end, updates are small, and the model is settling into a basin rather than exploring. Two things follow. First, whatever data the model sees during that final phase gets weighted disproportionately in the final weights — there is no subsequent large-LR phase to wash it out. Second, because the LR is small, high-quality data can be introduced without destabilising the run.

Put together: the last few percent of a run is unusually cheap leverage. Ai2 named this midtraining, and the numbers make the case. Olmo 3 7B trains 5.93T tokens of bulk pretraining, then 100B tokens — 1.7% of the run — on Dolmino, a mix of high-quality maths, science, code, instruction-following and reading-comprehension data sampled from a ~2.2T-token pool. Then 50B more on long documents. The last 2.5% of tokens are doing a strikingly large share of the work on exactly the capabilities benchmarks measure.

How midtraining mixes get built

The methodology here is worth knowing because it is unusually clean. Ai2 describes a two-part loop. Microannealing runs cheap, parallel ablations: take a mid-run checkpoint, anneal it on one candidate data source, measure. Because each run is short and they are independent, dozens of sources can be evaluated at once. Then integration tests combine the sources that looked promising into candidate mixes and run a full annealing pass to check that the gains compose rather than cancel.

That distinction — parallel per-source feedback, then centralised composition checks — is the general shape of every serious data workflow. Per-source signals are cheap and misleading in aggregate; composition is expensive and is where the surprises live.

The failure mode: last-mile overfitting

The uncomfortable flip side of “the end of training is high-leverage” is that the end of training is also where you can most cheaply teach a model to look good. Anneal on data that resembles your evaluation suite and scores rise without the underlying capability moving. There is no bright line separating “curriculum design” from “training on the test distribution,” and the honest position is that midtraining mixes are chosen against benchmarks and should be understood as partially benchmark-fitted. Holding out a subset of evaluations that never informs any data decision is the only real defence, and Ai2 explicitly recommends the practice.

What is an epoch?

The textbook definition is one pass over the training set:

This quantity is nearly meaningless for a modern LLM, and it is worth being precise about why, because the reasons are also the reasons the corpus is hard to describe at all.

“Unique” depends on your dedup threshold. Two documents differing by one sentence: one document or two? MinHash at J = 0.7 says one; at J = 0.9 says two. The denominator is a function of a hyperparameter you chose.

Sources overlap. arXiv papers appear in the arXiv dump, in Common Crawl, in citing PDFs run through OCR, and in synthetic data generated from them. Per-source dedup does not catch cross-source duplication.

Mixture weights resample unevenly. With wmath = 0.08 on a maths pool that is 2% of the corpus, maths is traversed four times while web is traversed once. There is no single epoch count — there is one per domain, and the whole point of the mixture step is that they differ.

Synthetic data blurs the boundary. A model-generated rephrasing of a source document is neither a copy nor an independent sample. It is a lossy, high-variance re-encoding of the original with the generator's biases layered on. Counting it as a fresh unique token is wrong; counting it as a repeat is also wrong.

The right abstraction: effective unique tokens

The useful formalism comes from data-constrained scaling. Rather than counting exposures, model the value of the k-th exposure to a token as decaying geometrically. With U unique tokens and E epochs, the effective data is

and this D′ substitutes for D in the usual Chinchilla-style law. The parameter R*D is the half-life of repetition — the number of repeats after which a token has lost 1/e of its value. Fitted on runs up to 9B parameters and 900B tokens, it came out around 15.

The practical numbers that fall out of that fit are the ones worth memorising:

There is also a striking allocation consequence. Under data constraints, the compute-optimal move is to train smaller models for more epochs than a naive Chinchilla extrapolation would suggest — because extra parameters, like extra epochs, have decaying marginal value once the data is fixed. Both axes decay; the optimal allocation shifts accordingly.

Knowing whether it worked

Everything above is a sequence of decisions. Decisions require feedback, and the feedback loop for pretraining data is unusually bad: the thing you want to measure costs millions of dollars to produce, and the cheap proxies for it are noisy in ways that are easy to mistake for signal.

This section is the counterpart to all of the above. It is also, in practice, where most of the intellectual work in a pretraining effort goes.

Level 1: training loss

Available continuously, free, and only useful for detecting catastrophe. Training loss depends on the data distribution, so two runs on different mixtures have incomparable losses — a run on easier data will show lower loss and a worse model. Use it for loss spikes, divergence, and dataloader bugs. Do not use it to compare data recipes.

Level 2: held-out perplexity, on multiple sets

Fix a set of validation corpora, evaluate every run on all of them. The plural matters: a single held-out set from the same distribution as your training data measures fit, not generality. Olmo's setup evaluates against a multi-source perplexity bundle for exactly this reason — you want per-domain perplexities so you can see that a mixture change improved web at the cost of code.

Perplexity is genuinely predictive of downstream performance, following a rough power law, and it is the cheapest signal that transfers across mixtures. Its limitation is that it saturates: two good models can have nearly identical perplexity and quite different capabilities, because the differences live in the tail of the distribution that perplexity averages over.

Level 3: benchmarks — and the noise problem

Base models are evaluated with likelihood-scored multiple choice and few-shot completion: MMLU, ARC, HellaSwag, PIQA, WinoGrande, GSM8K, HumanEval, and so on. Two structural problems appear immediately.

Formulation is not standardised. Whether you score the full answer string or the letter label, how you normalise by length, how many shots, what the prompt template is — all of these move scores by several points, and different papers make different choices. OLMES exists to fix this: a fully documented, reproducible standard covering the choices that are usually left implicit, specifically so that small base models (which need the “cloze” formulation) can be compared meaningfully against larger ones (which can handle the natural formulation).

Small models are at chance on most of what you care about. The models you can afford to run ablations with — 150M to 1B parameters on 100B tokens — score at random on MMLU, GSM8K and most code benchmarks. So the evaluations that matter at 7B give you no signal at the scale where you are actually making decisions.

Level 4: signal, noise, and decision accuracy

This is the framework that makes the previous problem tractable, and it is the most useful idea in this section.

For any benchmark, define two quantities. Signal is the spread of scores across a population of models — a benchmark where everything scores 51–53% cannot separate anything. Noise is the standard deviation of the score across the final checkpoints of a single training run — a benchmark whose score wobbles by three points between adjacent checkpoints cannot certify a two-point improvement.

Their ratio predicts what you actually want to know: decision accuracy, the probability that the ranking you observe at small scale is the ranking you would observe at large scale. Empirically, across 225 models trained on 25 corpora, SNR at small scale correlates strongly with decision accuracy (R² ≈ 0.63), while signal or noise alone do not.

Three practical consequences follow directly:

• Compute SNR for every benchmark in your development suite and drop the noisy ones, or move them out of your headline aggregate so one erratic task cannot dominate it.
• Prefer aggregates over capability clusters to individual tasks — averaging suppresses per-task noise while preserving signal. Olmo 3's OlmoBaseEval is built on exactly this principle.
• When fitting task scaling laws to predict large-model performance, remember that the noise floor of your target bounds how accurate the prediction can possibly be. If the 13B model's own score wobbles by 1.5 points, a scaling law predicting it to within 1 point is not measurably better than one predicting it to within 1.5.

Level 5: predicting the run you can only do once

The endgame is to make the small-scale signal predictive rather than merely suggestive. Two complementary approaches are now standard.

Task scaling laws. Fit performance as a function of compute across a ladder of small models, then extrapolate to the target scale. Done per data recipe, this turns “which recipe wins at 1B?” into “which recipe's fitted curve is higher at 32B?” — a different and more relevant question, since curves can cross.

Empirical decision benchmarks. DataDecide's contribution was to actually measure how well small experiments predict large outcomes, across 25 corpora, model sizes up to 1B, and multiple seeds. Its headline finding is deflating and useful: the ranking of corpora at a single small size is already a strong baseline for predicting the best corpus at the target scale. The expensive scaling-law ladder often does not beat just training at 150M and looking.

The thing you must not skip: decontamination audits

Benchmark contamination is not a binary. It ranges from verbatim test items in the training set, through paraphrases and translations, to discussion of the benchmark's contents, to training on data selected for resemblance to the benchmark's distribution. Only the first is straightforwardly detectable by n-gram overlap, and only the first is unambiguously cheating.

Practical minimum: run n-gram decontamination against every evaluation you report, do it after data selection rather than before, report the removal rates, and hold out a set of evaluations that never informs any data decision at all. That held-out set is the only number in your report that is not, in some sense, fitted.

What to carry away

Seven things, in the order they bite.

1. The objective is one line; the distribution is the artefact. Every lab minimises the same loss. What differs is D, and D is manufactured. Treat the corpus as the model's real specification.

2. The pipeline is a chain of irreversible filters, and order matters. Extraction, language ID, heuristics, safety, dedup, classification, decontamination. Roughly 1–10% of raw crawl survives. Each step's failure mode is a class of text that silently disappears.

3. Deduplication is a privacy control before it is an efficiency measure — and duplication carries endorsement as well as redundancy, which is why the modern answer is dedup globally and then upsample by measured quality rather than by accidental copy count.

4. Quality is not a property of documents. It is resemblance to a positive set you chose. State the positive set and the token horizon or the claim is not falsifiable.

5. Mixture selection is a real optimisation problem with a nasty oracle, and every method — DoReMi, mixing laws, RegMix, Aioli — is a way of making that oracle cheap. The weights they produce do not transfer between pools; the methods do.

6. The end of training is cheap leverage and cheap self-deception. A 100B-token midtraining stage on 1.7% of the budget moves benchmarks more than any equivalently sized change earlier. That is exactly why it needs a held-out evaluation suite that never touched a data decision.

7. Measure your noise floor. Most published data-recipe comparisons cannot support the conclusions drawn from them, for the entirely fixable reason that nobody reported the standard deviation across the last thirty checkpoints of a single run.

The next article picks up where the base model stops: a network that has learned the shape of the corpus, and has never once been told whether any particular continuation was good.

References

Grouped by topic. Every entry links to its primary source.