The question you actually have
Nobody starts a pretraining run by asking “how good can a language model get?” The question is always narrower and more annoying than that. You have a cluster reservation. You have a date. Somewhere between those two facts is a fixed number of floating-point operations you are allowed to spend, and you have to decide, before you spend any of them, what shape of model to spend them on.
Make the model too big and you will run out of budget before it has seen enough text to fill its capacity. Make it too small and you will pour tokens into something that cannot absorb them. Both mistakes cost the same amount of money. Only one of them is visible in the loss curve while it is happening.
The remarkable thing — the thing this whole article is about — is that this decision is not a matter of taste. Under a set of assumptions that hold surprisingly well, the best allocation is the solution to a two-line constrained optimisation, and you can estimate its coefficients from runs that cost a thousandth of the real thing. That is the science. The rest of the article is about where the assumptions leak.
How to read the callouts
- Nuance
- A subtlety that changes how a result should be read, not whether it is true.
- Trap
- A conclusion the arithmetic invites but does not actually support.
- Approximation
- A number that is fitted, quoted, or estimated rather than derived.
- Aside
- A worked example or sanity check. Skippable.
Start by looking at the thing you are choosing between. Below is one fixed compute budget, drawn as the set of all models you could buy with it. Every point on that curve costs exactly the same. They do not perform the same.
Model Loss is evaluated with the Chinchilla parametric form L = E + A·N−α + B·D−β using the constants published in Hoffmann et al. (2022): E = 1.69, A = 406.4, α = 0.34, B = 410.7, β = 0.28. Those constants are themselves contested — §7 gets into that. Treat the shape as real and the third decimal place as decoration.
Approximation 00 — these constants do not reproduce Chinchilla, and that is deliberate
Put Gopher's budget of 5.76×1023 FLOPs into the figure above and it recommends about 32 billion parameters at roughly 93 tokens per parameter — not the 70 billion at 20 tokens per parameter that DeepMind actually trained. That is not a bug in the figure. Hoffmann et al.'s own third estimation approach reported about 40B for that budget while their first two reported 63B and 67B, and the published constants for that third approach were later shown to fit the underlying data poorly.
Every figure in this article uses those published constants, unmodified, so you can check the arithmetic against the paper. The shapes, the trade-offs and the exponents are all real; the absolute level of the token-per-parameter recommendation is exactly the quantity that §7 is about. Read the figures for structure, not for a number to put in a planning document.
Three things are worth noticing before we derive any of it.
The curve is a hyperbola, not a line. Doubling the model halves the tokens. That is not a modelling choice; it is what a fixed compute budget means, and it comes directly from the identity in the next section.
The loss curve is flat near its minimum. Over a factor of two or three in model size, the achievable loss barely moves. This is why every lab that has published an IsoFLOP analysis says some version of “the optimum is broad,” and it is why the flagship model of a given generation is usually a round number rather than the exact argmin.
The walls are steep and asymmetric. Going too large hurts faster than going too small, because the data term has the smaller exponent in this fit and therefore the slower decay. Under-training a big model is the more expensive mistake.
Three numbers and one identity
Scaling laws are written in three variables, and it is worth being pedantic about them because almost every published disagreement in this field turns out to be a disagreement about definitions rather than about nature.
These are not independent. Once you know the architecture, C is determined by N and D, and the constant that links them is almost exactly six.
Where the six comes from
Take a single weight matrix W with P entries and push one token's activation vector through it. Every weight participates in exactly one multiply and one accumulate, so the forward pass costs 2P floating-point operations. That factor of two is the whole story; everything after it is bookkeeping.
The backward pass computes two different things, and this is the part people misremember. It needs the gradient with respect to the input, so the error can keep flowing down the network — that is another matrix product of the same shape, another 2P. And it needs the gradient with respect to the weights, so the optimiser has something to work with — a third product, a third 2P. Backward is twice forward because it is two matrix products instead of one.
Derived The count is exact for the linear layers and ignores everything that is not a matrix product against a parameter: normalisation, activations, residual adds, softmax, and the attention score computation itself. Those are cheap in FLOPs and expensive in bandwidth — a distinction the training-step walkthrough takes apart properly.
Multiply by every parameter in the network and by every token you push through it, and you have the identity. Six FLOPs per parameter per token, D tokens, N parameters.
Nuance 01 — D is a throughput count, not a corpus size
Often said: “the model was trained on 15 trillion tokens.”
Actually: D in C = 6ND counts token-visits, not distinct tokens. A 4-trillion-token corpus seen three times is D = 12T for the purpose of computing C, and something rather less than 12T for the purpose of predicting loss. Section 9 is about exactly that gap. When a model card says “trained on X tokens” it is usually the throughput number, which is the one you want here and not the one you want there.
What the six leaves out
The identity is an approximation, and it is worth knowing in which direction it lies. Two costs are missing.
The first is the output head. Projecting the final hidden state onto a vocabulary of a hundred thousand tokens is a genuine matrix product, and if you count “parameters” as non-embedding parameters — as Kaplan et al. did — then you have silently dropped it. For a 100M-parameter model with a 50k vocabulary, the head is most of the model. For a 400B-parameter model it is a rounding error. That size-dependent bias is going to matter enormously in §7.
The second is attention itself. The QK⊤ and AV products touch no parameters at all; their cost scales with sequence length, not with model size. Per token per layer, forward and backward together, they cost roughly 12·d·S operations before the causal mask and about half that after it. Compare that to the 72·d² the layer's linear parts cost, and the ratio is S/(6d) — negligible at 2k context and a third of your budget at 8k context on a 4k-wide model.
Derived The reference bar counts the linear layers (72Ld²), the output head (6Vd), and causally-masked attention (6LdS). Kaplan's 6P uses non-embedding parameters only; Chinchilla's 6N counts embeddings as parameters but not as FLOPs; DeepSeek's M drops the vocabulary term and adds attention with the unmasked 12LdS coefficient. Assumes dffn = 4d and full multi-head attention.
Drag the width down toward 512 and watch the three conventions fan apart. That fan is not a curiosity. It is, as we will see, most of the reason the field spent two years believing a wrong answer.
Approximation 01 — six is a modelling constant, not a measurement
Nobody has ever measured exactly 6ND FLOPs on a real cluster. Recomputation from activation checkpointing adds a partial extra forward pass — typically 30–35% on top for full checkpointing, much less for selective. Mixed precision changes which operations are cheap without changing the count. And the number you divide by to get MFU is the same 6ND, which means MFU quietly inherits every one of these conventions. Two labs reporting “40% MFU” may not be reporting the same thing.
What a scaling law is
A scaling law is a claim of a very specific shape: that some measure of quality falls off as a power of some measure of resource, over many orders of magnitude, with no visible feature in between. Written out, the claim is
and the reason it is interesting is not that it fits — almost anything fits a curve with three free parameters — but that the same three parameters keep working two orders of magnitude past where you fitted them.
Why the log-log plot is the whole trick
Subtract the floor and take logs of both sides:
which is a straight line with slope −α. That is the entire diagnostic. If your loss data plotted against log-scale axes falls on a line, you have a power law and you can read the exponent off with a ruler. If it curves down, you are near an irreducible floor you have not subtracted. If it curves up, something is broken — usually your learning rate schedule, sometimes your data.
The floor L∞ is doing real work here and it deserves a name. It is the entropy of the text under the tokenisation you chose, plus whatever your architecture structurally cannot represent. You cannot train it away. And crucially, you cannot see it on a log-log plot of raw loss — it shows up only as a mysterious flattening, which is exactly what people mistake for “scaling is over.”
Model Synthetic data from the stated functional form with 1.5% multiplicative noise, so the fit behaves like a real one. The point of the last stage is not that extrapolation works — it is how sensitive the extrapolation is to a floor you guessed wrong.
Trap 01 — a good fit is not evidence of a law
Three free parameters will fit almost any monotone decreasing curve over one decade. What makes a scaling law load-bearing is out-of-sample behaviour: fit on small runs, predict a big one, check. Every serious scaling study in the reference list does this explicitly and reports the held-out error. A paper that reports only R² on the fitting set has told you nothing you can spend money on.
Kaplan: the first map
In January 2020, Kaplan and colleagues at OpenAI published the observation that language-model loss follows smooth power laws in model size, dataset size, and training compute, over more than six orders of magnitude, with the trends showing no sign of bending. They also reported that architectural details — depth versus width, attention head count, feed-forward ratio — mattered far less than the total parameter count, within very wide bands.
That second finding is the one that changed the field. It said: stop tuning the shape, start buying scale.
Three bottlenecks, one at a time
The clean way to understand their result is to think of three separate experiments, each of which removes two of the three constraints.
Parameter-limited. Give the model effectively unlimited data and train to convergence. Then loss depends only on capacity, and falls as a power of N. Kaplan reported an exponent around αN ≈ 0.076, which is small: you need roughly a 10,000× increase in parameters to halve the reducible loss.
Data-limited. Give the model effectively unlimited capacity and a fixed corpus, with early stopping. Loss falls as a power of D with exponent αD ≈ 0.095.
Compute-limited. Fix the FLOP budget, allocate it optimally, and loss falls as a power of C with exponent αC ≈ 0.050.
Reported Exponents are as published in Kaplan et al. (2020). The bottleneck map in the last stages is drawn from the additive two-term form and is a modelling device, not a reproduction of any figure in that paper.
The map in the last stage is the useful mental object. At any point in the (N, D) plane, one of the two reducible terms dominates. Along the diagonal ridge where they are comparable, you are balanced; everywhere else, one of your two resources is being wasted. Compute-optimal training is the statement that the budget line should cross that ridge.
Nuance 02 — the exponents are small, and that is the point
Often said: “scaling laws show that loss improves rapidly with scale.”
Actually: an exponent of 0.05 means a 10× compute increase buys you a factor of 100.05 ≈ 1.12 in reducible loss — about 12%. Scaling laws are a description of how slowly and how reliably things improve. The reliability is what makes them valuable; the slowness is what makes the industry's capital expenditure make sense only if small loss improvements translate into large capability improvements. Whether they do is §11.
The allocation problem, solved
Here is the whole thing in one page. Assume the two-term loss and the compute identity:
The constraint kills one variable. Substitute D = C/(6N) and the problem becomes one-dimensional:
Read that last line carefully, because it is doing more work than it looks like.
The exponents must sum to one, which is not a coincidence: it is forced by C = 6ND. If N scales as Ca, then D = C/(6N) scales as C1−a. Any paper reporting exponents that do not sum to one has either changed the compute convention or made an arithmetic error, and both happen.
An invariant worth memorising
Multiply the stationarity condition through by N and something clean falls out:
So “compute-optimal” does not mean the two error terms are equal. It means they are in the ratio β:α, weighted by how fast each responds to investment. The resource with the steeper exponent gets to carry more of the remaining error, because moving it is cheaper.
Derived Everything here follows from the two boxed equations above; only A, B and E are borrowed from Hoffmann et al. Set α = β and the tokens-per-parameter ratio becomes budget-independent — that is the exact condition under which a fixed number like “20” can be a constant at all.
Nuance 03 — “scale N and D equally” is a statement about α = β
Often said: “Chinchilla showed model size and tokens should be scaled equally.”
Actually: they should be scaled equally if and only if the two exponents are equal. Chinchilla's first two estimation approaches landed on a ≈ b ≈ 0.5, which is the equal-scaling regime. Their own parametric fit gave α = 0.34 against β = 0.28, which implies tokens-per-parameter grows as C0.097 — slowly, but not at zero. The famous ratio is a snapshot of a slowly moving quantity, evaluated at one budget in 2022.
Aside 01 — where 20 goes if you take the drift seriously
Take the drift exponent seriously for a moment, but anchor it on what was actually trained rather than on the parametric fit. Chinchilla ran at roughly C = 5.8×1023 FLOPs at 20 tokens per parameter. Llama 3's flagship budget was about 3.8×1025, a factor of 66 more. Under (α−β)/(α+β) = 0.097 the ratio should have moved to 20 × 660.097 ≈ 30. Meta's own IsoFLOP study, on their own corpus, put the optimum at 402B parameters and 16.55T tokens — a ratio of 41.
Two different labs, two different datasets, two different decades of hardware, and the answers land within a factor of 1.4 of each other. The slope is the robust part. Note also that the parametric fit's own prediction at Chinchilla's budget is a much higher ratio than 20, which is the discrepancy §7 takes apart — so this agreement is between the drift rate and the measurements, not between the fit's absolute level and anything.
Chinchilla: the correction
For two years after Kaplan, the field built models the way that paper implied: parameters first, tokens as an afterthought. GPT-3 was 175B parameters trained on 300B tokens — under two tokens per parameter. Gopher was 280B on the same 300B. Megatron-Turing NLG was 530B on 270B. Everyone had converged on roughly 300 billion tokens, and everyone was scaling the model.
In March 2022, Hoffmann and colleagues trained more than 400 models spanning 70M to 16B parameters on 5B to 500B tokens, and reported that all of those flagships were badly misallocated. Their headline: for compute-optimal training, model size and training tokens should be scaled at roughly equal rates. Every doubling of parameters should come with a doubling of tokens.
They then did the thing that makes the paper credible. They took Gopher's exact compute budget, re-spent it on a 70B model trained for 1.4 trillion tokens, and the smaller model won — broadly, and by a lot.
How the measurement is actually made
The method worth internalising is their second one, the IsoFLOP sweep, because it is the one you can afford to run yourself and it makes almost no assumptions about functional form.
Model A reconstruction of the procedure, not of the published numbers: losses are generated from the two-term form with noise so the fitting behaves realistically. The methodology — sweep N at fixed C, take the minimum of each valley, fit a power law through the minima — is exactly what Hoffmann et al. call Approach 2, and what Meta, DeepSeek and others have repeated on their own corpora since.
Three properties of this procedure are worth stating plainly.
It never assumes a loss law. The parabola through each valley is a local fit used only to locate a minimum. You could find the minimum by eye. The only extrapolation happens at the very end, on the sequence of minima.
It costs one full sweep per budget. A four-budget, six-model study is 24 pretraining runs. That is why scaling studies are done at 1018–1022 FLOPs and extrapolated to 1025, and why the extrapolation distance is the number to interrogate in any scaling claim.
The valleys get flatter as the budget grows. Meta observed this explicitly in the Llama 3 study, and used it as licence to round 402B up to 405B. Flat valleys are good news for engineering and bad news for measurement: the flatter the valley, the noisier the estimate of its minimum.
Approximation 02 — three approaches, three answers
Hoffmann et al. estimated the frontier three ways: minimum-over-training-curves, IsoFLOP sweeps, and a parametric fit to all runs. At Gopher's budget the three predicted roughly 67B, 63B and 40B parameters respectively. The paper trained 70B. The first two agree well; the third does not, and that gap is not cosmetic — it is the thread Epoch AI later pulled on.
Why the two disagreed
This is the most instructive episode in the entire scaling-law literature, and it is instructive for a reason that has nothing to do with language models. Two careful groups, running honest experiments, published exponents of 0.73 and 0.50 for the same quantity. At a budget of 1025 FLOPs those two rules differ by more than an order of magnitude in recommended model size. The disagreement was not about the world. It was about the measuring apparatus.
| Study setup | Model exponent | Counts output-head FLOPs? | Retunes small runs? | Practical conclusion |
|---|---|---|---|---|
| Original Kaplan fit | ≈ 0.73 | × | × | Spend compute mostly on a larger model |
| Chinchilla IsoFLOP frontier | ≈ 0.50 | ✓ | ✓ | Grow model size and training tokens together |
| Controlled reanalysis | ≈ 0.50 | ✓ | ✓ | The gap largely disappears under matched accounting |
Why the field changed course. The later recommendation did not win because a new law of nature appeared. It won because small and large runs were compared on a fairer footing: the compute ledger was consistent and each scale got suitable optimization settings. Once those controls were added, under-trained large models stopped looking artificially attractive.
Schematic The endpoints are published: Kaplan's a ≈ 0.73, Chinchilla's a ≈ 0.50, and Porian et al.'s finding that correcting three factors recovers a ≈ 0.5. Of the intermediate steps, only the first is quantified in that paper (counting head FLOPs “reduces the exponent by more than 0.1”); the sizes of the other two steps here are illustrative of direction and rough magnitude, not measured values.
The three corrections
The last layer's FLOPs went uncounted. Kaplan defined model size without embedding parameters — a defensible choice, made because it produced cleaner depth-independence. But because their embeddings and output head were tied, they also dropped the head from the FLOP count. Porian et al. measured the resulting bias: roughly 10% understatement at large model sizes, growing smoothly to roughly 90% at small ones. A size-dependent error in the x-axis is precisely the kind of error that rotates a fitted slope.
Warmup was too long at small scale. Kaplan used a warmup schedule sized for full-length training runs. In the compute-optimal regime, small models are supposed to train for very few tokens — sometimes fewer tokens than the warmup itself consumed. Those runs were therefore systematically handicapped, which makes small models look worse than they are, which pushes the fitted optimum toward large models.
Learning rate and batch size were not re-tuned per scale. Optimal learning rate and batch size both move with scale. Holding them fixed while sweeping model size measures “models at their tuned optimum” at exactly one point and “models at somebody else's optimum” everywhere else. Porian et al. also found that AdamW's β2 matters specifically at small batch sizes — a detail with no business being load-bearing in a paper about scale, and yet.
Correct all three and the Kaplan setup reproduces the Chinchilla exponent. A separate analysis by Pearce and Song argues that a large part of the gap is attributable to the parameter-accounting choice alone, combined with the small scales at which the original analysis was performed. The two explanations are compatible; they emphasise different pieces of the same accounting problem.
Trap 02 — Chinchilla is not settled either
In 2024, Epoch AI attempted to replicate the third Chinchilla estimation approach from the paper's own plots. They found the reported constants fit the reconstructed data poorly, disagree with the paper's other two approaches, and carry confidence intervals so tight they would require hundreds of thousands of runs rather than a few hundred. The likely causes were an optimiser that stopped short of convergence during the fit and rounding in the reported exponents. Their re-derivation lands back near equal scaling — consistent with Approaches 1 and 2. A 2025 follow-up by Schaeffer and colleagues then stress-tested the whole prescription, found that the parameter definition alone can move things by up to 15%, and concluded that despite all of it, the practical guidance still holds. That is roughly where the field sits: the recipe survives, the third decimal place does not.
Nuance 04 — the lesson generalises past scaling laws
Every ingredient of this dispute was a hyperparameter that varied with scale and was held fixed anyway. Warmup, learning rate, batch size, β2, and the definition of the x-axis. If you run your own scaling study — and §10 argues you should — the single highest-value thing you can do is re-tune, at every scale, everything you would otherwise have inherited from the run before. DeepSeek's contribution here was to publish scaling laws for the hyperparameters themselves before fitting anything about model size.
Train-optimal is not deploy-optimal
Chinchilla answers a specific question: given a training budget, what is the lowest loss you can reach? That is not the question a company asks. A company asks: given a quality target and a product that will serve a few billion requests, what is the cheapest model to own?
Those are different optimisations and they have different answers. Llama 3's 8B model was trained on more than 15 trillion tokens — nearly two thousand tokens per parameter, roughly a hundred times the Chinchilla ratio. Nobody at Meta was confused. They were optimising a different objective.
The lifetime objective
Write the total compute a model will ever consume as training plus serving. Inference is a forward pass only, so it costs about 2N per token rather than 6N:
Now the problem inverts. Instead of fixing compute and minimising loss, you fix a loss target and minimise compute. The set of models reaching that target is a contour in the (N, D) plane; you want the cheapest point on it.
Run the Lagrange conditions and one line tells the whole story. Minimising 6ND + 2NT subject to A N−α + B D−β = ℓ − E gives
Set T = 0 and this collapses back to αTN = βTD — the training-optimal invariant from §5. Turn T up and the left side grows, which forces N−α up, which means N down. Inference demand pushes the optimum toward smaller models, monotonically, with no exceptions and no free parameters.
Model Iso-loss contours from the Hoffmann parametric form; cost is 6ND + 2NT with no allowance for KV cache, batching efficiency, quantisation, or the fact that a smaller model may serve at a much better tokens-per-second. All of those push the same direction, so the figure understates the effect.
Sardana and colleagues did this properly in 2024, extending the Chinchilla formulation to include inference cost and solving it for real dollar prices as well as raw FLOPs. Their conclusion: anyone anticipating on the order of a billion requests should train models smaller and longer than Chinchilla-optimal. They also trained 47 models to check, and found quality still improving at ratios as extreme as ten thousand tokens per parameter.
Nuance 05 — extreme over-training breaks the law you fitted
Often said: “the scaling law says more tokens keep helping, so just train longer.”
Actually: Sardana et al. also ablated their own fitting procedure and found that coefficients estimated only from runs at conventional token-per-parameter ratios overestimate the value of additional tokens out at extreme ratios. If you intend to train at 1,000 tokens per parameter, you need scaling data collected at 1,000 tokens per parameter. This is the same disease as §7 in a new costume: fitting in one regime and spending in another.
Aside 02 — the other reasons to over-train, which are not in any equation
Fitting in one GPU's memory is a step function, not a gradient. So is fitting in a laptop's. Latency budgets are set by product managers, not by loss curves. A model half the size has half the KV cache, roughly twice the achievable batch size, and a serving cost that falls faster than N because of how badly small batches use a GPU — a point the serving playbook makes at length. None of this appears in 6ND + 2NT, and all of it points the same way.
Not every token is worth one token
Everything so far has treated D as if tokens were fungible and infinite. They are neither. High-quality text is a finite resource, and the moment your token budget exceeds your corpus you start repeating — at which point D in the compute identity and D in the loss law stop being the same number.
Muennighoff and colleagues measured the gap directly, training more than 400 models up to 9B parameters and 900B tokens, with repetition counts ranging up to 1,500 epochs. Their headline result is unusually actionable:
Up to about four epochs, repeated data is nearly as good as fresh data. An 8.7B model trained for four epochs on 44B unique tokens finished within half a percent of the validation loss of a single-epoch run on 178B unique tokens. That difference does not show up in downstream evaluations at all.
Meaningful gains continue to roughly sixteen epochs, then collapse. Beyond that the return on additional compute decays toward zero: you are spending FLOPs to memorise text you have already extracted everything from.
Model Uses the data-constrained functional form of Muennighoff et al. (2023), D′ = U + U·R*·(1 − e−R/R*) with R* ≈ 15.4 for tokens, where R is the number of repetitions past the first pass. The saturation ceiling of (1 + R*)·U is a property of that fit, not a measured asymptote — nobody has trained 1,500 epochs at frontier scale to check.
The practical consequence is that the compute-optimal calculation splits into two regimes. Below your corpus size, use the frontier from §5. Above it, the token axis starts costing more than it returns, and the optimum shifts back toward more parameters and more epochs than the naive rule suggests. Muennighoff et al. show exactly this: in the data-constrained regime, new compute should go into both parameters and epochs, but with the token axis discounted.
Trap 03 — “we are running out of data” is a claim about a specific corpus
The four-epoch result means a 4T-token corpus supports something like a 16T-token training run at almost no penalty, and a 60T-token run at a real but non-fatal one. The binding constraint on modern pretraining is much more often quality filtering and deduplication policy than raw availability — Qwen3 reports 36 trillion tokens across 119 languages, assembled partly from PDF extraction and synthetic generation, which is a statement about pipeline engineering rather than about the internet running dry. Where that data comes from and how it is filtered is the subject of the next article in this series.
Predicting a big run from small ones
All of the above is only useful if you can estimate the coefficients before you spend the money. You can, and the procedure has become standardised enough that three independent labs have published nearly the same recipe.
Reported Budget ranges are as published: DeepSeek swept eight budgets from 1017 to 3×1020 FLOPs; Meta swept 6×1018 to 1022 with models from 40M to 16B; Gadre et al. built 104 models from 11M to 6.9B across three data distributions. The layout of the grid is schematic.
Fit the hyperparameters before you fit the model
DeepSeek's scaling study is the most useful one to copy, because it does the thing §7 says everyone should have done from the start: it establishes scaling laws for the optimiser settings first, so that every model in the sweep is measured at its own optimum rather than somebody else's. They report
and only then fit the allocation. Their second contribution is a change of variable that has quietly become standard: instead of parameters, use non-embedding FLOPs per token, which they call M, so that C = M·D exactly. With that definition they report
Note what M buys. It absorbs the vocabulary question, the attention-versus-linear question, and eventually the dense-versus-sparse question, because it is defined by arithmetic rather than by a parameter count. If you take one implementation lesson from this article, take that one.
Nuance 06 — the exponents depend on your data, not just on transformers
DeepSeek re-ran their allocation study on a higher-quality corpus and found the model-scale exponent rose while the data exponent fell: better data means you should spend relatively more of a fixed budget on capacity. In the language of §5, improving your corpus raises β relative to α. This is why borrowing another lab's exponents is a mistake even when their architecture matches yours — you have a different β because you have a different dataset.
How far can you extrapolate?
Further than you would guess, if you are careful about what you are predicting. Gadre and colleagues built a testbed of 104 models and fitted laws that extrapolate in both model size and degree of over-training. They then predicted the validation loss of a 1.4B-parameter model trained on 900B tokens — 32× past compute-optimal — from experiments costing 300× less compute, and got it. Meta extrapolated across roughly three and a half orders of magnitude, from a study topping out at 1022 FLOPs to a run at 3.8×1025.
Trap 04 — the extrapolation that fails is the one that changes something else too
Scaling studies extrapolate well when the only thing changing is scale. They fail when the large run also introduces a new data mix, a new context length, a new architecture, a new precision, or a new parallelism strategy that changes the effective batch size. Every one of those is a change of law, not a change of scale. The discipline is boring and absolute: whatever you sweep at small scale must be the only thing different at large scale.
From loss to capability
Everything so far predicts one number: held-out cross-entropy. Nobody ships cross-entropy. The quantity that decides whether a model was worth training is its behaviour on tasks, and the relationship between the two is real, monotone, and considerably looser than the scaling-law literature's tidy plots suggest.
Schematic Curves are illustrative of the qualitative behaviour reported by Gadre et al. and by the Llama 3 scaling section; the individual-benchmark traces are synthetic. The structural claims — loss smooth, single benchmarks jagged, many-task averages smooth again — are the reproducible ones.
Two published workflows handle this, and they are worth contrasting.
Gadre et al. go direct. They fit a relation from perplexity to average top-1 error across a large task suite, then use it to predict downstream error for models 20× larger than anything in the fit. The averaging is doing essential work: the relation holds for the mean over many tasks, and is much weaker for any single one.
Meta go in two hops. First correlate compute-optimal models' negative log-likelihood on the downstream task itself against training FLOPs; then correlate that task-specific NLL against task accuracy, borrowing the Llama 2 family as extra anchor points at higher compute. The intermediate variable is the trick: task NLL is smooth where task accuracy is not, because it is a continuous readout of the same underlying competence.
Nuance 07 — pretraining loss is not a universal ordering
Often said: “lower pretraining loss means a better model.”
Actually: lower loss on the same validation distribution means a better model on that distribution. Two models with equal loss on different corpora are not comparable at all, and the same is true across tokenisers — cross-entropy per token depends on how many tokens your tokeniser spends on the same text, which is why bits-per-byte exists and why several scaling studies report it instead. Comparing published loss numbers across labs is almost always meaningless.
Nuance 08 — the mapping is architecture-dependent
Work on sparse models has found that dense and mixture-of-experts models with the same pretraining loss do not always land in the same place on downstream tasks. The loss-to-capability map has its own dependence on how the parameters are arranged, which means a scaling study on dense models does not automatically transfer its downstream predictions to a sparse one even when its loss predictions transfer fine.
Do scaling laws ever break?
Two claims are in tension, both supported by careful work, and the resolution is more interesting than either.
The first: in 2022 Wei and colleagues catalogued a set of abilities that appear abruptly with scale — near-chance performance across several orders of magnitude of compute, then a sharp rise. They called these emergent, and defined the term operationally: not present in smaller models, present in larger ones, and not predictable by extrapolating from the smaller ones.
The second: in 2023 Schaeffer, Miranda and Koyejo argued that much of this is manufactured by the measuring instrument. If your metric is discontinuous or strongly nonlinear in the underlying per-token competence, then a perfectly smooth improvement in that competence will look like a phase transition. Change the metric on the same model outputs and the sharp transition can disappear.
You can construct the effect from scratch in about two lines, which is what the next figure does.
Derived Per-token accuracy is a smooth power law by construction; every other curve in the figure is an exact function of it. No claim is made here about which real abilities are or are not genuinely discontinuous — the figure shows only that a smooth substrate is sufficient to produce the appearance of emergence under a common scoring rule.
What the argument does and does not establish
It establishes that sharpness in a benchmark curve is not, by itself, evidence of anything sharp happening inside the model. Three separate mechanisms are enough: a metric that multiplies per-token probabilities, a metric with a hard threshold, and a test set too small to resolve accuracies below its own granularity. Schaeffer et al. showed that changing the metric while holding the model family fixed removes the effect on many of the catalogued tasks.
It does not establish that nothing discontinuous ever happens. The strong version — “emergence is always an artefact” — is not what the paper claims and is not supported. A model either can or cannot do modular arithmetic on unseen operands, and the transition between those states is a real event in the world even if a well-chosen metric renders the curve smooth on the way in.
Nuance 09 — the practical reading, for someone planning a run
If you need to forecast whether a planned model will clear a capability bar, do not extrapolate the accuracy curve. Find the continuous quantity underneath it — per-token likelihood on the task, an edit distance, a log-probability margin — extrapolate that, and threshold at the end. This is the same manoeuvre Meta uses in the two-stage forecast from §11, and it is the only part of the emergence debate you actually have to act on.
Breaks that are not artefacts
Separately from the metric question, there is the empirical fact that some scaling curves genuinely change slope. A single power law cannot represent that. Caballero and colleagues proposed broken neural scaling laws: a functional form built from several power-law segments joined by smooth transitions, which can fit non-monotonic and inflected behaviour that a single power law cannot, and which extrapolates better on curves that have a break in them.
Schematic The break here is synthetic. The point is structural: a fit that looks excellent on everything you have measured constrains the next decade of scale only if you have assumed there is no break in it, and that assumption is not testable from inside the fitting window.
Trap 05 — a break is not automatically bad news
The conversation about “scaling walls” assumes breaks go one way. They do not. A break to a steeper slope is a regime where scale suddenly buys more than it used to; a break to a shallower one is the wall people worry about. Both are visible only in hindsight, and neither is predicted by a functional form fitted before the break. The honest position is that a single power law is a hypothesis about the absence of structure, and the absence of structure has held for a remarkably long time without anyone being able to say why.
What is N, anyway?
Everything above assumed that a model has one number called its size. For a dense transformer that is true. For a mixture-of-experts model it is not, and the ambiguity is large enough to flip the headline conclusion about any given model.
DeepSeek-V3 has 671 billion total parameters and activates about 37 billion per token, trained on 14.8 trillion tokens. Divide by the total and you get 22 tokens per parameter — almost exactly Chinchilla-optimal. Divide by the active count and you get 400 — heavily over-trained. Both numbers are correct. They answer different questions.
Reported Total and active counts for the named models are as published in their technical reports. The tokens-per-parameter columns are arithmetic on those figures. Nothing here claims which column is the right one to use — the argument of this section is that the question is genuinely open.
Which count belongs in which equation
The compute side has a clean answer. Training FLOPs depend on what the hardware executes, so C ≈ 6 Nactive D, and DeepSeek's M — non-embedding FLOPs per token — is exactly the right variable because it is defined by arithmetic rather than by storage. Adopt M and the dense-versus-sparse question stops corrupting your compute axis entirely.
The capacity side does not have a clean answer. A model's ability to store and retrieve knowledge plausibly tracks total parameters; its ability to compute per token plausibly tracks active ones. Neither Ntotal nor Nactive substituted into A·N−α reproduces observed behaviour across sparsity levels, which is why the sparse scaling literature adds sparsity as its own axis.
The results there are still consolidating, but the direction is consistent: Clark and colleagues found routed models gave efficiency advantages that faded at large scale; Krajewski and Ludziejewski and colleagues added dataset size and expert granularity and found the advantage persists when granularity is treated as a variable; Frantar and colleagues derived exponents that interpolate between dense and sparse regimes; Abnar and colleagues at Apple charted optimal sparsity directly and reported that at a fixed FLOP budget, raising sparsity while growing total parameters improves pretraining loss — with the optimal sparsity rising as the budget grows.
Nuance 10 — “is this model over-trained” is not answerable for an MoE without saying which N
When you read a comparison of tokens-per-parameter across models, check whether sparse models are being counted on the total or the active axis, and whether the same choice was applied to all of them. Mixing conventions inside one table is common and makes the table meaningless. The safest presentation reports both columns and lets the reader pick, which is what the figure above does.
Aside 03 — the same disease, one level down
Notice that this is §7 again. Kaplan and Chinchilla disagreed because “model size” had two definitions that diverged systematically with scale. Dense and sparse comparisons disagree because “model size” has two definitions that diverge systematically with sparsity. In both cases the fix is the same: define the axis by what the hardware does, not by what the checkpoint contains.
The planner, and what to carry
Here is the whole article as a calculator. Put in a cluster and a schedule; it applies §2 to get a budget, §5 to allocate it, §9 to check the allocation against your corpus, and §8 to adjust for serving. Every number it prints has been derived somewhere above.
Model Peak throughput figures are vendor specifications for dense BF16 tensor-core arithmetic, tagged as such because nobody reaches them. The allocation uses the Hoffmann constants throughout; substitute your own the moment you have run the ladder in §10, because §10's whole point is that these are somebody else's dataset's exponents.
What to carry away
The budget is a curve, not a point. C = 6ND turns a cluster reservation into a one-dimensional choice, and the identity is worth deriving once so you know which costs it omits and in which direction.
The allocation is closed-form, and its exponents depend only on the ratio α:β. Two labs can share a policy and disagree completely about achievable loss. Constants set the intercept; only the exponent ratio sets the slope.
Twenty is not a constant. It is a horizontal line through one measurement, valid exactly when the two loss exponents are equal, and nothing published pins them to that precision.
Train-optimal is the wrong objective for anything you plan to serve. The lifetime-cost optimum moves monotonically toward smaller models with more tokens, and this follows from the sign of one term rather than from any empirical finding.
Tokens stop being fungible after about four passes, and the compute identity keeps charging you for them long after the loss law stops paying out.
The failures are accounting failures. Every documented case of a scaling law being wrong — Kaplan's exponent, the Chinchilla parametric fit, mixed dense-and-sparse comparisons — came from a definition that drifted systematically with the very variable being swept. Bias that grows with scale does not look like noise. It looks like a slope, and a slope is what you extrapolate.
Forecast continuous things. Loss extrapolates; accuracy does not. Whether emergence is real or an artefact of scoring is a question you can mostly route around by predicting the smooth quantity underneath and thresholding at the end.
The next article in this series takes the token axis seriously and asks where 15 trillion tokens actually come from, how they are filtered, and what a pretraining curriculum looks like once you stop pretending D is a single number.
References
Grouped by topic. Every entry links to its primary source. Where a number in this article is contested, the entry that contests it is listed alongside the one that reported it.
The founding results
- Deep Learning Scaling is Predictable, Empirically · Hestness et al., 2017 The power-law observation, three years before it had a name, across machine translation, language modelling, image classification and speech.
- Scaling Laws for Neural Language Models · Kaplan et al., 2020 The paper that made scale a strategy. Source of the three exponents in §4 and of the claim that architectural shape matters far less than total size.
- Training Compute-Optimal Large Language Models · Hoffmann et al., 2022 Chinchilla. Over 400 models, three estimation approaches, and the parametric constants used throughout the figures in this article.
Auditing and reconciling those results
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models · Porian, Wortsman, Jitsev, Schmidt, Carmon, 2024 The three corrections in §7: last-layer FLOPs, warmup duration, scale-dependent optimiser tuning. Also finds that careful learning-rate decay is not essential to the Chinchilla result.
- Reconciling Kaplan and Chinchilla Scaling Laws · Pearce and Song, 2024 Argues that much of the gap follows from Kaplan counting non-embedding rather than total parameters, at the small scales where that distinction is largest.
- Chinchilla Scaling: A Replication Attempt · Besiroglu, Erdil, Barnett, You (Epoch AI), 2024 The published Approach 3 constants fit the reconstructed data poorly and carry implausibly narrow confidence intervals. Their re-derivation lands back near equal scaling.
- Evaluating the Robustness of Chinchilla Compute-Optimal Scaling · Schaeffer et al., 2025 Finds the parameter definition alone can move results by up to 15%, and concludes that the practical prescription survives anyway.
Past the training-optimal regime
- Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws · Sardana, Portes, Doubov, Frankle, 2024 The lifetime-cost formulation of §8, solved for FLOPs and for dollars, validated on 47 models out to ten thousand tokens per parameter.
- Language Models Scale Reliably with Over-Training and on Downstream Tasks · Gadre et al., 2024 104 models across three data distributions; predicts a 32×-over-trained run from experiments costing 300× less, and relates perplexity to average downstream error.
- Scaling Data-Constrained Language Models · Muennighoff et al., 2023 The four-epoch and sixteen-epoch results, and the effective-token functional form used in Figure 10.
Scaling studies run for production models
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism · DeepSeek-AI (Bi et al.), 2024 Hyperparameter scaling laws fitted before the allocation study, the non-embedding-FLOPs-per-token variable M, and the finding that better data shifts compute toward model size.
- The Llama 3 Herd of Models · Grattafiori, Dubey et al. (Meta), 2024 An IsoFLOP study from 6×1018 to 1022 FLOPs extrapolated to 3.8×1025, and the two-stage downstream forecast described in §11.
- DeepSeek-V3 Technical Report · DeepSeek-AI, 2024 671B total parameters, 37B active, 14.8T tokens — the worked example of §13's ambiguity.
- Qwen3 base model cards · Qwen Team, 2025 36 trillion pretraining tokens across 119 languages, with scaling-law-guided hyperparameter tuning done separately for dense and sparse variants.
- Gemma 3 Technical Report · Gemma Team, Google DeepMind, 2025 Per-size token budgets that scale with model size rather than staying fixed — a small, explicit statement of the over-training policy.
Emergence and its critics
- Emergent Abilities of Large Language Models · Wei et al., 2022 The original catalogue, with the operational definition: not present in smaller models, present in larger ones, not predictable by extrapolation.
- Are Emergent Abilities of Large Language Models a Mirage? · Schaeffer, Miranda, Koyejo, 2023 The metric argument reconstructed in Figure 13, including the demonstration that changing the scoring rule on fixed outputs can remove the transition.
- Broken Neural Scaling Laws · Caballero, Gupta, Rish, Krueger, 2022 A functional form of several power-law segments joined smoothly, able to fit inflections and non-monotonic behaviour a single power law cannot.
Sparse and architecture-dependent scaling
- Unified Scaling Laws for Routed Language Models · Clark et al., 2022 The first systematic treatment of routed models, and the finding that routing's efficiency advantage faded at the scales they could reach.
- Scaling Laws for Sparsely-Connected Foundation Models · Frantar, Riquelme, Houlsby, Alistarh, Evci, 2023 Exponents that interpolate between dense and sparse regimes, with sparsity as an explicit variable.
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models · Abnar et al. (Apple), 2025 Charts optimal sparsity directly, and notes that dense and sparse models at matched pretraining loss can differ downstream.
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient · Ludziejewski et al., 2025 Joint laws over dense and sparse models with expert granularity treated as a variable, extending the fine-grained MoE line of work.
Companion articles on this site
- Anatomy of a Training Step · ML systems from first principles Where 6ND comes from at the level of individual kernels, and the memory ledger that decides whether a given N is trainable at all.
- The Serving Playbook · ML systems from first principles Why the inference term in §8 is more forgiving to small models than 2N per token suggests.
- Why Overparameterized Models Generalize · Understanding LLMs, 02 · forthcoming The prior question: why a model with more parameters than data points does not simply memorise.