Why Transformers Do Not Explode

01 — The question, stated precisely

A 100-layer Transformer is a composition of two hundred nonlinear maps. Nothing in the architecture forbids the forward activations from growing by a factor of 1.5 per layer, which after 100 layers is a factor of 4 × 10¹⁷. Nothing forbids the backward gradients from shrinking by 0.7 per layer, which after 100 layers is 3 × 10⁻¹⁶ — comfortably underflow in fp16. And yet these models train. Not effortlessly, and not without a decade of accumulated tricks, but they train.

This piece is about why. It is a single continuous argument, and the argument has a shape worth stating up front, because most treatments split it into two disconnected topics — "initialization" and "training stability" — that are really the same topic examined at two different timescales.

The shape is this. At step zero, the network is a random function, and whether it is a usable random function is a question about variance propagation: does a signal entering the bottom of the stack arrive at the top with roughly the scale it started with, and does an error signal entering the top arrive at the bottom the same way? That is the initialization question, and it is answerable in closed form. After step zero, gradient descent begins moving the weights, and every quantity the initialization analysis fixed starts to drift. Attention logits grow. The residual stream grows. Second moments in the optimizer go stale. Weight norms creep upward and the effective learning rate creeps downward. The initialization question becomes a control question: what mechanisms keep those drifting quantities inside the range where the arithmetic still works?

So the honest answer to "why doesn't it explode" is not one mechanism. It is a stack of them, each catching a different failure, and each one is legible once you know what it is catching. Here is the stack, in the order we will build it:

Two of these — normalization geometry and optimizer mechanics — have their own dedicated treatments elsewhere on this site, and I will lean on them rather than re-derive them. The Geometry of Normalization covers what LayerNorm and RMSNorm actually do to a vector; What an Optimizer Actually Does covers Adam's moment estimates in detail. Here I care only about how those objects behave as parts of the stability stack.

02 — Part I: Variance through a matrix multiply

Part I

Start with the smallest possible object. One linear map, no bias, no nonlinearity:

Assume the input coordinates are centred and identically scaled, and that the weights are drawn independently of the input with zero mean:

Take one output coordinate. It is a sum of n_in products:

Because each term has zero mean, the mean of the sum is zero. For the variance, use the fact that for independent zero-mean a and b, Var(ab) = 𝔼[a²b²] − 𝔼[ab]² = 𝔼[a²]𝔼[b²] = Var(a)Var(b). If we additionally treat the n_in terms as uncorrelated, variances add:

That is the whole of Part I in one line. A matrix multiply multiplies the variance of its input by n_in · σ_w². Call that quantity the layer's gain:

Everything downstream follows from asking what happens when you multiply by g a hundred times, and from noticing that the backward pass has its own gain which need not equal the forward one.

The gain, seen

The backward pass has its own gain

Gradient descent needs the derivative of the loss with respect to the input, so it can hand it to the layer below. For y = Wx:

This is the same calculation with the matrix transposed, so the fan-in of the backward multiply is n_out, not n_in:

And here is the tension that generates every classical initialization scheme. You would like both gains to equal one:

These are simultaneously satisfiable only when n_in = n_out. For a rectangular layer you must choose, or compromise.

Xavier / Glorot: split the difference

Glorot and Bengio's answer was to take the harmonic compromise — set the variance so that the two constraints are averaged rather than one being satisfied exactly:

In uniform form, since a uniform on [−a, a] has variance a²/3, this is the familiar U(−√(6/(n_in+n_out)), +√(6/(n_in+n_out))). The derivation is not deeper than the two variance equations above: they conflict, so you average them. When n_in = n_out the compromise is exact and both gains are 1.

Xavier's stated derivation assumes the activation function is approximately linear near the origin with unit slope — which is true of tanh at small inputs, and was the standard nonlinearity when the paper was written. That assumption is the thing ReLU breaks.

Kaiming / He: pay for the half the ReLU throws away

ReLU zeroes half of its input. If z is symmetric around zero with variance σ_z², then a = max(0, z) has

Note this is the second moment, not the variance — post-ReLU activations are not zero-mean, and the He derivation deliberately tracks 𝔼[a²] rather than variance, which is the right quantity to propagate through the next linear layer. So the nonlinearity contributes a gain of exactly ½, and the layer's total forward gain is ½ · n_in · σ_w². Setting that to one:

That factor of 2 is the entire content of Kaiming initialization. It is compensation for the half of the signal ReLU discards. The same logic gives the general rule: for any nonlinearity φ, compute its gain γ = 1 / 𝔼[φ(z)²] under a unit-variance Gaussian input, and set σ_w² = γ / n_in. This is exactly what PyTorch's calculate_gain table encodes: √2 for ReLU, √(2/(1+a²)) for leaky ReLU with slope a, 5/3 for tanh, 1 for linear and sigmoid.

03 — Part II: Stacking L of them

Part II

Now compose. A plain deep network — no residuals, no normalization — computes

Variances multiply, so the whole-network gain is the product of the per-layer gains:

Take logs and the structure becomes obvious:

The log-variance performs a random walk with drift down the depth of the network. The drift is L · 𝔼[log g]. If the mean log-gain is even slightly positive, the variance grows exponentially in L; slightly negative, it decays exponentially. There is no third option — g = 1 exactly is a measure-zero knife-edge, and finite-width fluctuations mean that even a perfectly tuned σ_w gives you a per-layer gain that jitters around 1.

The same product appears, transposed and reversed, in the gradient:

So exploding activations and vanishing gradients are the same phenomenon observed from opposite ends. This is the classical dichotomy:

Dynamical signal propagation: what "criticality" means precisely

The variance argument tracks one number per layer. A finer analysis tracks how two different inputs' representations relate as they go up the stack — because a network whose layers map every input to the same vector has destroyed the information the task needs, even if the variance is perfectly preserved.

The mean-field theory of Poole, Schoenholz and collaborators does exactly this. For a random network with weight variance σ_w² and bias variance σ_b², two inputs with correlation c at layer ℓ have correlation c' = C(c) at layer ℓ+1, where C is a fixed map determined by σ_w, σ_b and the nonlinearity. Iterating C down the depth drives c to a fixed point c*, and the rate of approach is governed by the slope χ₁ = C′(c*):

χ₁ is the same object as the backward gain — it is the per-layer multiplier on the gradient. So the two pictures agree: the ordered phase is the vanishing-gradient phase, the chaotic phase is the exploding-gradient phase, and criticality is the boundary. The mean-field framing adds the crucial extra fact that the depth over which information survives scales like 1/|log χ₁|, which diverges at the critical point. A network initialized off criticality has a finite "depth scale" beyond which it is, informationally, a constant function.

Orthogonal initialization and dynamical isometry

There is a stronger condition than "the average gain is 1". Look at the end-to-end input–output Jacobian J = ∂x_L/∂x_0. Preserving the mean squared singular value of J at 1 is the variance condition. Dynamical isometry demands more: that the entire singular value spectrum of J concentrate near 1, so that no direction in input space is amplified or attenuated relative to any other.

Gaussian random matrices cannot do this. The product of many i.i.d. Gaussian matrices has a singular value spectrum that spreads out without bound as depth grows, even at criticality — the mean is controlled but the tails are not. Orthogonal matrices can: an orthogonal matrix has all singular values exactly 1, and products of orthogonal matrices are orthogonal. Saxe and colleagues showed that orthogonal initialization gives depth-independent learning times in deep linear networks; Pennington and colleagues extended the analysis to nonlinear networks and showed that the achievable isometry depends jointly on the weight ensemble and the nonlinearity — ReLU cannot achieve dynamical isometry at all, while tanh with orthogonal weights can. The most striking demonstration is Xiao and colleagues training a 10,000-layer vanilla CNN with no residual connections and no normalization, purely by initializing for isometry.

04 — Part III: Residual streams change everything

Part III

The residual block replaces composition with accumulation:

This is a small syntactic change with a large structural consequence, and it is worth being precise about what changes, because "residuals help gradients flow" is repeated far more often than it is derived.

What happens to the magnitude of the stream

Suppose F_ℓ at initialization produces an output roughly uncorrelated with its input, with variance σ_F², and suppose the stream has variance σ_ℓ². Then because the cross term has zero expectation:

Variances add rather than multiply. If every branch contributes the same σ_F² = σ_ℓ² · β in the pre-norm case where F sees a normalized input, the recursion is σ_{ℓ+1}² = σ_ℓ²(1 + β) — geometric again. But if the branch output has a fixed absolute variance v (which is what a normalized-input branch with fixed weights delivers), then

This is the single most important structural fact about the residual stream, and it is a completely different growth law from the plain stack. Plain networks grow like g^L. Residual streams grow like √L. Exponential becomes square-root. A hundred layers turns a potential 10¹⁷ into a factor of 10.

Why gradients flow: the identity term is unconditional

Differentiate the residual update:

and chain it across the whole stack:

Expand the product. You get a sum over every subset S ⊆ {ℓ,…,L−1} of the ordered product of the J_k for k in S:

Three things fall out of this expansion.

First, the identity survives. The S = ∅ term is exactly I, with no factors at all. So no matter how small the J_k are — even if every branch is dead — the gradient reaching layer ℓ is at least the gradient at the top, unattenuated. In the plain stack that guarantee does not exist: if the gains multiply to 10⁻¹⁶, the bottom of the network receives 10⁻¹⁶. The residual connection converts a multiplicative bound into an additive one. This is the actual mechanism behind "residuals help gradient flow", and it is worth stating in exactly those terms: the gradient's lower bound is no longer a product.

Second, the network is implicitly an ensemble of paths of many depths. There are 2^L−ℓ terms in that sum, one per subset, and the term with |S| = k passes through exactly k nonlinear branches. Veit and colleagues made this concrete: a residual network behaves much like an ensemble of relatively shallow networks, and the gradient contribution is dominated by the short paths, because the long ones involve products of many small Jacobians. The measured effective depth of a very deep ResNet is far shorter than its nominal depth.

Third, the Jacobian is not conditioned on the branch being useful. Balduzzi and colleagues studied the shattered gradient problem: in plain deep ReLU nets, the spatial correlation structure of the gradient decays exponentially with depth, so the gradient starts to look like white noise and first-order methods lose their footing. In residual networks with normalization, that correlation decays as a power law instead. Depth stops destroying the gradient's usable structure.

05 — Part IV: Where the normalizer goes

Part IV

Residual connections fix the multiplicative gradient problem but leave the stream growing like √L, and they say nothing about the scale a branch's input arrives at. Normalization handles both, and where you put it relative to the residual add turns out to be one of the most consequential single decisions in Transformer design.

The two options:

The original Transformer used Post-LN. Essentially every large language model since roughly 2020 uses Pre-LN or a variant. Understanding why is a matter of asking, for each, what the identity path looks like.

Post-LN destroys the clean identity path

In Post-LN, the normalizer sits outside the add, so it is on the identity path. The gradient must pass through LN at every layer, and the LN Jacobian is not the identity. Xiong and colleagues derived the key scaling: for LayerNorm,

The gradient through a LayerNorm is inversely proportional to the norm of its input. This is the mechanism behind Post-LN's characteristic pathology, and it explains a lot at once:

• If the model takes a large update early and the inputs to LN grow, the gradient through every LN shrinks proportionally. The model becomes hard to move, exactly when it has moved somewhere bad.
• Xiong and colleagues showed that at initialization, Post-LN's expected gradient norm at the last layer scales like O(d√(ln d)) independent of depth, while the gradients at different depths are badly unbalanced — the top layers get a much larger share.
• Their conclusion — that Pre-LN Transformers have well-behaved gradients at initialization while Post-LN does not — is the reason warmup was practically mandatory for Post-LN and became optional for Pre-LN.

The DeepNet authors traced the full causal chain in a 18-layer encoder-decoder and it is worth reproducing because it is a genuinely good piece of debugging: Post-LN takes an exploding update at the very start of training; this lands it in a bad region; the inputs to LN become large; by the √d/‖x‖ relation, the gradient through LN collapses; the model can no longer escape. Instability and vanishing gradients are the same event seen at two moments. They also showed that better initialization alone does not fix it — a Post-LN model with down-scaled lower layers still had a larger last-layer gradient norm, yet converged, which is what ruled out "exploding gradients in deep layers" as the root cause and pointed at the update magnitude instead.

Pre-LN keeps the identity path clean

In Pre-LN the normalizer is inside the branch. The identity path from x_ℓ to x_{ℓ+1} is untouched, so the whole (I + J) expansion from Part III applies verbatim, with J = ∂F(LN(x))/∂x. The gradient reaching layer 1 is bounded below by the gradient at layer L regardless of what the branches do. That is why Pre-LN trains at depth without warmup and tolerates much larger learning rates.

Pre-LN has its own price, and it is not merely cosmetic:

• Gradient imbalance the other way. Because the stream's magnitude grows with depth while the branch outputs do not, the relative contribution of each branch shrinks with ℓ. Deep layers modify a large stream by a small amount. Shleifer and colleagues observed that Pre-LN gradients at the bottom layers tend to be larger than at the top, the mirror image of Post-LN's problem, and this shows up as a consistent quality gap relative to a converged Post-LN model.
• Effective depth erosion. Push this far enough and the top layers of a very deep Pre-LN model are doing almost nothing — the identity dominates. Recent work has given this the name "curse of depth", arguing that in Pre-LN models the output variance grows with depth such that deep blocks' Jacobians approach the identity and the layers contribute little.
• The stream still grows. Pre-LN normalizes the branch input, not the stream. You still need a final LN before the unembedding, and the stream's √L growth is real. Massive-activation phenomena — a handful of dimensions carrying enormous magnitude in the residual stream of trained LLMs — live in this regime.

06 — Part V: Scaling the residual branch

Part V

The interventions in this section all do the same thing from different angles: they make the per-block contribution shrink as the network gets deeper, so that the total change the network can make in one update does not grow with L.

Why the update magnitude, not the gradient magnitude, is the right target

Here is the argument in its cleanest form, following DeepNet. Consider what one SGD step does to the network's output:

For an N-layer Post-LN Transformer with standard initialization, DeepNet's Theorem 4.2 gives

where v and w stand in for the value and output projection scales of each sublayer, and α is the coefficient on the residual term. In vanilla Post-LN, α = v = w = 1, so ‖ΔF‖ = O(Σ_i ‖Δθ_i‖) — the update magnitude accumulates linearly in the number of sublayers. Doubling the depth doubles how far the function moves per step, at fixed learning rate. That is the exploding-update problem, and it is a depth problem that has nothing to do with the gradient being large.

This immediately explains two things that were previously folklore. Warmup works because it shrinks ‖Δθ‖ during the window when the model is most fragile. Small initialization works because it shrinks √(v² + w²). They are two ways of attacking the same product.

DeepNorm: fix the product, not the factors

DeepNorm makes both moves at once and picks the constants so the bound is Θ(η) independent of depth. The block becomes

with α > 1 up-scaling the residual term, and the branch weights down-scaled by β < 1 at initialization. Setting α² = (2N)^{1/2} and v² + w² = (2N)^{−1/2} with v = w = β gives, for a decoder-only stack of M layers:

For encoder-only stacks the formulas are the same with N; for encoder–decoder the encoder gets α = 0.81(N⁴M)^{1/16}, β = 0.87(N⁴M)^{−1/16} and the decoder α = (3M)^{1/4}, β = (12M)^{−1/4}, because cross-attention propagates the encoder's magnitude into the decoder and the decoder therefore needs a tighter bound. Only the feed-forward weights and the value and output projections of attention are scaled by β; the query and key projections are left at gain 1, because — by DeepNet's Lemma 4.1 — softmax output is a convex combination of the value rows, so W_Q and W_K do not change the magnitude bound of the attention output at all. They change which values you average, not how big the average is.

The result is a Post-LN model that trains at 1,000 layers. DeepNet reports 100L-100L encoder–decoder reaching 28.9 BLEU on WMT-17 En-De where vanilla Post-LN diverges past 18L-18L, and a 200-layer 3.2B-parameter multilingual model outperforming a 48-layer 12B baseline by roughly 5 BLEU across 7,482 translation directions. The depth scaling they observe is logarithmic: L(d) = A·log(d) + B.

The family of branch-scaling tricks

DeepNorm is the most carefully derived member of a family that all share the same idea. It is worth seeing them together because in practice you will meet several of them in the same codebase:

The one you will actually see in most open LLM code is the first: initialize residual output projections at σ/√(2L). The factor of 2 is because each block has two residual branches (attention and MLP). It is the cheapest possible version of the DeepNorm argument — no architecture change, one line in the init function — and it captures most of the benefit for the depths people actually train at.

07 — Part VI: Attention has its own failure mode

Part VI

Everything so far applies to any deep residual network. Attention adds a failure mode that is specific to the softmax, does not show up in the variance analysis at all, and has become the dominant practical instability at large scale.

The mechanism, step by step

Attention logits are inner products of learned projections:

Now watch what training does. Nothing in the loss penalizes ‖W_Q‖ or ‖W_K‖. Nothing normalizes q or k. The 1/√d_h factor is a constant chosen for the initialization scale of q and k, and it does not adapt. So over tens of thousands of steps:

Once the logits are large, softmax(z) is numerically one-hot. Two bad things follow, and they are distinct.

Entropy collapse. The attention distribution has essentially zero entropy. Every query attends to exactly one key. Zhai and colleagues named this and proved a tight lower bound on attention entropy that decreases exponentially fast with the spectral norm of the attention logits — so once the weight matrices' spectral norms start growing, entropy has very little room left. Empirically they found low attention entropy tracks training instability across image classification, self-supervised learning, translation, speech and language modelling: oscillating loss, then divergence.

The softmax Jacobian dies. This is the part that turns a representational problem into an optimization problem. For p = softmax(z):

The head is now frozen in a configuration it cannot escape, because the very thing that would tell it to change — the gradient through the softmax — has been zeroed by its own sharpness. This is a positive feedback loop with a self-locking end state, and it is why the failure is abrupt rather than gradual.

It is worth being precise about the timing, because the two failures are not simultaneous and the gap between them matters. Entropy is a log-scale quantity and starts falling as soon as the logits have any spread at all; p_i(1−p_i) is a product and stays near its uniform value until one probability is genuinely close to 1. Concretely, a head whose entropy has fallen from 2.77 nats to 0.05 — already attending to essentially one key — still has a softmax Jacobian only about six times smaller than a uniform head's. The gradient does not vanish until max_i p_i is within roughly 10⁻³ of 1. So there is a long window in which the head has stopped doing anything useful but is still, in principle, correctable, followed by a short window in which it locks. Figure 09 lets you sweep through both. Neither transition registers in the loss.

The fixes, and what each one actually constrains

QK normalization. Apply LayerNorm or RMSNorm to the queries and keys before the dot product:

Now ‖q‖ and ‖k‖ are pinned to √d_h by construction, so the logit magnitude is bounded by d_h/√d_h = √d_h regardless of how large the weights grow. The technique is due to Henry and colleagues, and it came to wide attention when Dehghani and colleagues hit divergence training ViT past 8B parameters and traced it to exactly this: extremely large attention logits producing near-one-hot attention weights with near-zero entropy. QK-norm fixed it. Wortsman and colleagues then showed the same instability reproduces in small models at high learning rate, and that qk-layernorm stabilizes training across three orders of magnitude of learning rate variation — which is what made it cheap to study and cheap to justify. It is now standard in OLMo 2, Gemma 3, Qwen 3 and much else.

Logit soft-capping. Squash the logits through a scaled tanh instead of normalizing their inputs:

This is smooth, keeps the gradient alive (unlike a hard clamp), and hard-bounds the logit at ±c. Gemma 2 shipped this with c = 50 on attention logits and c = 30 on the final output logits. The cost is that the derivative sech²(z/c) shrinks as z approaches the cap, so you are trading a hard failure for a soft one. It is also awkward to fuse with FlashAttention, which is part of why Gemma 3 moved to QK-norm instead.

z-loss. A different target: the output logits, not the attention logits. Add an auxiliary penalty on the log-partition function:

This keeps the softmax normalizer near 1 so the output logits stay near the log-probabilities. PaLM introduced it for exactly this reason, and Wortsman and colleagues confirm it as one of the two interventions (with qk-layernorm) that most reduce learning-rate sensitivity. It is cheap and it is essentially free insurance.

σReparam. Zhai and colleagues attack the spectral norm directly, reparameterizing every linear layer as a spectrally normalized matrix times a learned scalar. Since their entropy bound is in terms of the spectral norm of the logits, bounding it bounds the collapse. This is a stronger and more expensive intervention, and it let them train a ViT with no warmup, no weight decay, no LayerNorm and no adaptive optimizer — which is a useful demonstration of how much of the standard recipe exists to suppress this one failure.

QK-clip. The newest variant in this family monitors the attention logits during training and clips the weights whenever the logits exceed a threshold — a control loop rather than a static architectural constraint.

08 — Part VII: The optimizer is part of the dynamical system

Part VII

All of the above analyses treat the update as "gradient times learning rate". Adam does not do that. It divides by a running estimate of the gradient's second moment, and that division is a stateful, low-pass-filtered thing that can be wrong in ways that produce updates of arbitrary size from perfectly ordinary gradients.

Warmup: what the first thousand steps are for

Recall Adam's update:

At t = 1, v̂₁ = g₁² exactly, so Δθ₁ = −η · sign(g₁) — a full-magnitude step in every coordinate, based on a single minibatch. The bias correction makes the estimator unbiased, but unbiased is not the same as low-variance: with β₂ = 0.95 the effective averaging window is about 20 samples, and early on you have fewer than that. Liu and colleagues' RAdam analysis frames this precisely as the adaptive learning rate having excessive variance in the early phase, and shows that warmup functions as a variance-reduction heuristic.

Composing this with the DeepNorm bound from Part V gives the complete picture. The model's output moves by ‖ΔF‖ ≈ Σ_i ‖Δθ_i‖, which grows with depth; Adam's early ‖Δθ‖ is at its maximum and its noisiest; therefore the first few hundred steps of a deep Post-LN model are the single most dangerous window in the entire run. Warmup shrinks η exactly there. Standard practice for large models is a linear ramp over 1–4k steps or roughly 0.1–1% of total tokens.

Adam moments and the stale-second-moment blow-up

The subtler failure runs the other way. Suppose a parameter's gradient has been tiny for a few thousand steps — a rare token's embedding row, say, or a specialist expert in an MoE. Then v̂ ≈ small. Now a batch arrives containing that token. The gradient is normal-sized, but the denominator is stale:

Adam's normalization, which usually caps the step at roughly η, produces a step orders of magnitude larger. Molybog and colleagues formalized this as a theory of Adam instability in large-scale training, arguing that the correlation between m and v breaks down for time-domain-correlated gradients and that this produces the characteristic spike. The empirical evidence for the mechanism is strong: OPT-175B's logbook and the GLM-130B and BLOOM reports all describe spikes traced to optimizer state, and the standard mitigations are all about the denominator — raise ε (many large runs use 1e-8 → 1e-6 or higher), lower β₂ from 0.999 to 0.95 so the estimate forgets faster, or skip the offending batch entirely.

ε is not a numerical safety pad. It is a floor on the denominator, and it sets the largest possible update: |Δθ| ≤ η·|m̂|/ε. Raising it from 1e-8 to 1e-6 lowers the worst-case step by 100×. That is a real hyperparameter with a real trade-off, not a rounding guard.

Gradient clipping

Global-norm clipping, from Pascanu and colleagues:

Two properties worth being clear on. It preserves direction exactly — it is a projection onto a ball, not a per-coordinate clamp, so it does not distort which way you step. And it is a global norm, computed across all parameters, which in a distributed run means an all-reduce; per-parameter or per-layer clipping is a different and generally worse operation.

The right value of τ is empirical and should be chosen by looking at the histogram of ‖g‖ over a few thousand steps. τ = 1.0 is the near-universal default. The failure mode to watch for is clipping firing on most steps, which means it has stopped being a safety net and become an unannounced learning-rate schedule. If your clip rate is above a few percent, τ is wrong or something else is.

Weight decay, and the effective learning rate you didn't set

Weight decay interacts with normalization in a way that surprises people. For any parameter W that feeds directly into a normalization layer, the function is scale-invariant: LN(cWx) = LN(Wx). So the loss does not care about ‖W‖. But the gradient does — for scale-invariant W, ‖∂L/∂W‖ ∝ 1/‖W‖. That means the effective learning rate on the direction of W is η/‖W‖², and weight decay, by shrinking ‖W‖, silently raises it.

Two practical consequences. First, use AdamW, not L2-in-the-loss: Loshchilov and Hutter's point is that adding λ‖θ‖² to the loss makes the decay pass through Adam's 1/√v̂ denominator, so parameters with small gradients get decayed differently from parameters with large ones — which is not what you asked for. Decoupled decay applies θ ← θ(1 − ηλ) directly. Second, be careful what you exclude. Biases, gains and — importantly — token embeddings are usually excluded. OLMo 2 explicitly does not decay the embeddings, on the grounds that shrinking embedding norms amplifies early gradients through the subsequent normalization.

Batch size, gradient noise, and what changing it does

The gradient of a minibatch of size B is an average of B independent estimates, so

Two rules of thumb follow, and they disagree. The linear scaling rule — Goyal and colleagues — says scale η ∝ B, on the grounds that k steps at batch B ought to equal one step at batch kB. The square-root rule says scale η ∝ √B, keeping the ratio of update magnitude to gradient noise fixed, which is the right rule when the optimizer is normalizing by gradient magnitude the way Adam does. In practice: linear scaling with SGD in the small-batch regime, closer to square-root with Adam and at large batch. Both break down past the critical batch size, beyond which extra examples buy almost nothing.

The stability-relevant point is that these two rules differ by √B, so a team that scales batch by 4× and learning rate by 4× when they should have used 2× has doubled their learning rate without meaning to. Mid-run batch size changes — common in curriculum or sequence-length ramps — are a classic source of spikes for exactly this reason.

09 — Part VIII: Instability that lives in the number format

Part VIII

Every argument above assumed real arithmetic. The hardware does not do real arithmetic, and a substantial fraction of production training failures are failures of the number format rather than of the model.

The three formats, and the one number that separates them

The number that matters is the exponent width, because it sets the dynamic range, and dynamic range is what gradients need. FP16 and BF16 are both 16 bits, but BF16 has FP32's exponent field and FP16 does not. That single difference is why BF16 became the default for large model training: BF16 cannot overflow at any value FP32 could hold, so the whole apparatus of loss scaling becomes unnecessary. You pay for it in mantissa bits — BF16 has 8 bits of relative precision versus FP16's 11 — and the accumulation of rounding error is the price.

The three ways low precision kills a run

Overflow. Anything above 65,504 becomes inf in FP16, and inf − inf or 0 × inf becomes NaN, which then propagates to every parameter through the optimizer state in a single step. Attention logits are the classic culprit: an unbounded inner product of two growing vectors, computed before the softmax's max-subtraction can save you if the intermediate is materialized in FP16. This is the direct connection between Part VI and this section — attention logit growth and FP16 overflow are the same bug at different scales, and QK-norm fixes both.

Underflow. Gradients are small. The gradient of a well-scaled loss with respect to a deep parameter routinely sits around 10⁻⁷ to 10⁻⁹, which is below FP16's smallest normal value of 6.1 × 10⁻⁵, into the subnormal range or straight to zero. Micikevicius and colleagues measured this directly and found a large fraction of activation gradient values in FP16 training fall below the representable range and become zero — which is why the mixed-precision recipe exists at all.

Rounding stagnation. The quiet one. If θ ≈ 1 and Δθ ≈ 10−5, then in FP16 — whose spacing near 1 is about 10⁻³ — the sum θ + Δθ rounds back to θ. The update is silently discarded. Nothing overflows, nothing NaNs, the loss just stops improving. This is the reason for master weights: keep an FP32 copy of the parameters, apply updates to it, and cast down to FP16/BF16 only for the forward and backward passes. It costs 4 bytes per parameter and it is not optional.

Loss scaling

The fix for underflow is to move the gradients into the representable range before they get there. Multiply the loss by a constant S; by linearity every gradient is multiplied by S; unscale by 1/S before the optimizer step:

Dynamic loss scaling automates the choice of S: start high, and whenever a step produces inf or NaN, skip that step and halve S; after a few thousand clean steps, double it. The scale factor settles into whatever the run can tolerate. Note that a skipped step is data thrown away — a healthy run skips a fraction of a percent, and a run that skips 5% of steps has a real problem being masked by the scaler.

BF16 makes all of this unnecessary, which is why the modern default is: BF16 compute, FP32 master weights, FP32 optimizer state, FP32 reductions for anything summed across many terms — loss accumulation, the softmax denominator, the norm statistics, the gradient all-reduce.

FP8, and where the frontier is

FP8 has 3 or 2 mantissa bits, so per-tensor scaling is no longer sufficient — the dynamic range within a single tensor exceeds what the format can hold. The standard approach uses E4M3 for forward-pass tensors (weights, activations — precision matters more, range matters less) and E5M2 for gradients (range matters more), plus per-tensor scale factors maintained from a history of observed maxima. DeepSeek-V3 pushed further with fine-grained tile- and block-wise quantization and periodic FP32 accumulation inside the matmul, and trained a 671B-parameter MoE in FP8 end to end — the first widely documented FP8 training of a frontier-scale model.

What stays in higher precision in these recipes is instructive, because it is exactly the list of things this article has been about: normalization statistics, the softmax, the attention logits, the residual accumulation, the optimizer moments, and the gradient reduction. Everything that is a sum over many terms or a ratio that can be near zero.

10 — Part IX: A debugging taxonomy for loss spikes

Part IX

Everything above becomes actionable at the moment you are staring at a training curve that just went vertical. The single most useful thing to have is a decision procedure rather than a list of possible causes, because the causes are distinguishable and each has a signature that costs nothing to log.

Three notes on using this.

Replay is the highest-value single test. It cleanly separates data causes from state causes, and it costs one forward-backward pass. Everything else in the tree is an inference from a correlation; replay is an experiment.

The concrete data failure is more common than people expect. OLMo's team traced repeated gradient-norm spikes in OLMo-0424 to repeated n-grams in web data — sequences like a long run of a repeating base64 fragment — and filtering them was one of the explicit stability interventions in OLMo 2, alongside a mean-zero σ = 0.02 initialization, RMSNorm, reordered normalization and QK-norm. That is a useful data point about where the effort actually goes: half the stability work in a modern open recipe is architectural and half is dataset hygiene.

Recovery strategy matters as much as diagnosis. PaLM's team observed spikes that did not reproduce on replay, and concluded the cause was the interaction of specific data with a specific optimizer state; their fix was to restart from a checkpoint roughly 100 steps before and skip a few hundred batches. Skipping data is a legitimate intervention. So is lowering the learning rate for a window. What is not legitimate is doing either without logging it, because it silently changes the run you think you did.

11 — Part X: Making hyperparameters survive a scale-up

Part X

One last failure, and it is the most expensive kind, because it does not look like a failure. You tune a learning rate on a 100M model, scale to 30B, and the run is stable but worse than it should be. Nothing spiked. You simply used the wrong learning rate, and you will never know by how much.

The question is: can hyperparameters tuned on a small model transfer predictably to a much larger one? Under standard parameterization, no — and the reason connects directly back to Part I.

Consider what happens to a single pre-activation as width grows. Under standard parameterization with Kaiming init, σ_w² = 2/n, so the pre-activation variance is width-independent at initialization — that part is fine. But the update is not. A gradient step changes each of the n incoming weights by roughly the same amount, and those n changes add coherently in the pre-activation, because after one step of training the weight update is correlated with the input. So the change in a pre-activation from one step scales like n · Δw rather than √n · Δw. What was O(1) at initialization becomes O(n) after training starts. The optimal learning rate therefore has to shrink as width grows, and it shrinks at a rate you have to discover empirically for every new size.

Maximal Update Parametrization (μP) is a specific assignment of per-tensor multipliers, initialization variances and learning-rate scalings — different for input, hidden and output weights — chosen so that activations, their updates, and the logits are all Θ(1) in width. Yang and Hu derived it as the unique parameterization admitting maximal feature learning in the infinite-width limit; the practical payoff is μTransfer: parameterize the target model in μP, tune on a small proxy, and zero-shot transfer.

The headline results are worth quoting because they set the value of the technique: transferring pretraining hyperparameters from a 13M-parameter proxy beat published BERT-large (350M) numbers at a total tuning cost equivalent to pretraining BERT-large once; transferring from a 40M proxy beat published 6.7B GPT-3 numbers at 7% of the pretraining cost. What transfers is not only the learning rate — initialization scale, the cross-entropy temperature and the schedule shape were all reported roughly stable across widths from 128 to 4,096.

12 — What actually keeps it alive

Pulling the whole argument back together. A hundred-layer Transformer does not explode because eight independent mechanisms are each holding a different failure closed, and every one of them is doing real work:

1. Initialization sets the nominal gain to 1. σ_w² = γ/n_in, with γ the nonlinearity's gain. This makes step zero survivable. It does not survive step one thousand on its own.
2. Residual connections convert a product into a sum. The identity term in (I + J) means the gradient's lower bound stops being a product of L factors. Growth becomes √L instead of g^L.
3. Normalization pins the branch input. Pre-LN keeps the identity path clean at the cost of a gradient imbalance and eroding effective depth; Post-LN is better when it converges and needs DeepNorm to converge at depth.
4. Branch scaling bounds the per-step function change. Whether via 1/√(2L) output init, LayerScale, or DeepNorm's derived α and β, the target is ‖ΔF‖ = Θ(η) independent of L.
5. QK-norm and logit caps stop the softmax from saturating. This is the failure that scales with training time rather than with depth, which is why it was discovered late and hurts most at frontier scale.
6. Warmup, ε, β₂, and clipping tame the optimizer's own dynamics. The first thousand steps and the stale-second-moment blow-up are two ends of one problem: Adam's denominator is an estimate, and estimates are wrong in characteristic ways.
7. Master weights, BF16, and selective FP32 keep the arithmetic honest. Sums over many terms and ratios near zero stay in high precision; everything else can be cheap.
8. A scale-consistent parameterization keeps the hyperparameters correct. Not a stability mechanism so much as insurance against a failure that never announces itself.

If you take one operational thing from this: the cheap logs are the whole game. Per-layer activation RMS, per-layer max attention logit, global gradient norm, clip rate, loss-scale value, √v̂ per parameter group, and per-rank loss. Every failure mode in this article announces itself in one of those seven traces, usually thousands of steps before it announces itself in the loss. A run with those plots is debuggable. A run with only a loss curve is a mystery, and mysteries at 10,000 GPUs are expensive.

13 — References

Grouped by the part of the argument they support. Where a paper's own numbers appear in the text above, they are the ones reported in that paper; the figures use schematic values unless a caption says otherwise.