The Geometry of Normalization

BatchNorm was sold in 2015 as a cure for internal covariate shift. That explanation has not survived contact with the evidence. What survived is more interesting, and it is what actually motivates every modern design decision.

Story 1 — Internal covariate shift (mostly dead)

The original claim: as earlier layers update, the distribution of inputs to later layers keeps shifting, so later layers spend their capacity chasing a moving target. Normalizing pins the distribution and lets each layer learn against a stable input.

Santurkar et al. (2018) tested this directly by injecting deliberate, time-varying non-zero-mean noise after the BatchNorm layer — recreating covariate shift on purpose. The network trained just as well. Worse for the theory: they measured the distributional drift in networks with and without BN and found BN networks did not obviously have less of it. The mechanism named in the title is not the mechanism doing the work.

Story 2 — A smoother optimization landscape (alive)

The replacement explanation: normalization improves the Lipschitzness of the loss and of its gradients. Gradients become more predictive — the direction you compute at your current point stays useful over a longer step. That is exactly what buys you larger stable learning rates and less sensitivity to initialization.

Story 3 — Scale invariance and the implicit learning rate (very alive)

This is the one to keep in your head, because it explains weight decay interactions, learning-rate schedules, and why removing norms is hard.

Put a norm immediately after a linear layer. Then scaling that layer's weights by any k>0 changes nothing in the forward pass — the norm divides it right back out:

So the weight's magnitude is a free parameter that the forward pass ignores but the gradient sees. The effective step size in angular terms is η / ‖W‖². Two consequences you have certainly felt:

• Weight decay becomes a learning-rate control. Decay shrinks ‖W‖, which raises the effective LR. That is why weight decay on normalized layers behaves nothing like classical L2 regularization, and why the two are entangled in every modern schedule.
• Norm growth is the thing to monitor. Weight norms drift up during training, effective LR drifts down, and you get the "automatic LR decay" that normalized nets exhibit even with a flat schedule.

There is a fourth, quieter reason that dominates at scale and rarely gets a section in a paper: normalization keeps activations inside the representable range of bf16. Several of the most consequential design changes in the last three years — QK-Norm, norm reordering, sandwich norms — were adopted not for accuracy but because a 34B model was producing numbers a 16-bit float could not hold.

Take a tensor. Decide which axes you reduce over to compute the statistics. That single decision names your normalization layer. Everything below is the same four lines of arithmetic pointed at different slices.

Two conventions collide in the literature, so let's fix both. Vision people write (N, C, H, W) — batch, channels, spatial. Sequence people write (B, T, D) — batch, tokens, features. They map onto each other: C ↔ D is the feature axis, H·W ↔ T is the "positions" axis. The atlas below uses a single block so you can see both at once.

The one-line taxonomy

Say the tensor is x[n, c, s] — sample n, feature/channel c, position s. Then:

The right-hand column is the practical dividing line. BatchNorm is the only one whose forward pass depends on the other examples in your batch. That single property generates every headache in §04, and it is why it lost the transformer era outright.

A normalization layer is: centre, scale, then undo both with learned parameters. The last part sounds circular. It isn't — and understanding why is the difference between using these layers and designing with them.

Why the affine map back? Because the normalization step is a hard constraint the network never asked for. Forcing every feature to unit variance destroys information about relative feature importance that later layers may want. γ and β hand that expressiveness back — the layer can learn to partially undo normalization, or to re-express a feature at whatever scale is useful, while the gradient still flows through a well-conditioned operator.

Two properties fall out of the algebra and are worth memorizing:

• Re-scaling invariance: Norm(ax) = Norm(x) . Both LayerNorm and RMSNorm have it. This is the property that produces the effective-LR story in §01.
• Re-centring invariance: Norm(x + c1) = Norm(x) . LayerNorm has it. RMSNorm does not. That is the entire difference between them, and Zhang & Sennrich's 2019 argument was that re-centring contributes little to quality while costing a full pass over the vector.

The gradient is the interesting part

Normalization's backward pass is not element-wise. Because μ and σ depend on every element, the gradient w.r.t. any one input has three terms:

Read the bracket geometrically: the incoming gradient gets projected orthogonally to two directions — the all-ones vector 1 (that is the − mean(g) term, removing the component that would only shift the mean) and the current normalized activation x̂ (that is the third term, removing the component that would only rescale it). Both directions are ones the forward pass ignores, so their gradient is discarded.

Three consequences that matter in practice:

• The 1/σ prefactor means a layer whose activations shrink gets louder gradients. This is a stabilizing negative feedback loop — and it is exactly the mechanism recent work argues that "massive activations" exploit as a gradient attenuator (§11).
• RMSNorm drops the − mean(g) term (no centring), so its backward is genuinely cheaper, not just its forward.
• The projection means normalization layers destroy gradient information in two directions per layer. In a 100-layer network that is 200 discarded directions. Usually irrelevant; occasionally the explanation for a strange rank-deficiency you're debugging.

BatchNorm is still the right answer for a ResNet on ImageNet with batch size 256. It is the wrong answer for nearly everything you are training now, and the reasons are worth spelling out because they generalize.

The mechanism has a split personality. In training it uses the statistics of the current mini-batch. In evaluation it uses running estimates accumulated by exponential moving average. These are two different functions. The gap between them is where BatchNorm's failures live.

The five failure modes

1. Small batches. Estimator variance goes as 1/N. Detection and segmentation at batch 2 per GPU is where GroupNorm was born.
2. Variable-length sequences. Padding contaminates the statistics; masking them out makes the effective N position-dependent. Late positions in a batch of mixed lengths get statistics from a handful of examples. This is the structural reason transformers use LayerNorm.
3. Train/eval mismatch. Any distribution shift between training batches and inference inputs corrupts running stats. Fine-tuning a backbone at a new resolution silently invalidates them, which is why FrozenBatchNorm2d exists in every detection codebase.
4. Information leakage across examples. The prediction for example i depends on examples j ≠ i. Fatal for contrastive learning (the model can cheat by reading batch statistics — hence SimCLR's care, MoCo's shuffling BN, and BYOL's dependence on it), and awkward for anything with per-example guarantees.
5. Distributed training. Statistics are per-device by default, so your effective batch is the local batch, and changing your parallelism changes your model's function. SyncBN fixes it with an all-reduce of ∑x and ∑x² per layer per step — correct, and a throughput disaster at scale.

The BatchNorm variants worth knowing

• SyncBN — cross-device statistics via all-reduce. Correct, slow, still standard in detection.
• Ghost BatchNorm — deliberately compute statistics over sub-batches (e.g. 32 within a batch of 1024) to retain the regularizing noise that large-batch training destroys.
• Batch Renormalization — corrects the train/eval function gap by applying an affine correction toward the running statistics during training, with clipped correction factors.
• Frozen BN — statistics and affine fixed from pretraining. The default when fine-tuning a CNN backbone at small batch, and the reason so many detection models ship with BN in eval mode permanently.
• PowerNorm — Shen et al.'s attempt to rescue batch statistics for transformers: relax zero-mean to quadratic mean and use running statistics in the forward pass with a correction. Interesting; never displaced LayerNorm.

One property BatchNorm has that the per-sample norms do not: it decorrelates the batch dimension as a regularizer. The noise from mini-batch estimation is a genuine stochastic regularizer, and part of why removing BN from a ResNet costs accuracy even when training is stable. Ghost BN exists to preserve exactly that.

Forget the formulas for a moment. Geometrically, LayerNorm and RMSNorm each map an arbitrary vector onto a fixed-radius sphere. They differ in one step: LayerNorm first flattens the vector onto the hyperplane where the coordinates sum to zero. RMSNorm skips that.

So LayerNorm's image is a sphere of dimension D−2 living inside the hyperplane 1ᵀx = 0; RMSNorm's image is the full (D−1)-sphere. LayerNorm throws away one degree of freedom — the component along 1 — and RMSNorm keeps it. That is the whole difference, before the learned γ.

Does removing the mean matter?

Empirically: rarely, at scale, in decoder-only language models. That is why essentially every LLM after LLaMA uses RMSNorm. But the reasons it can matter are worth knowing.

• Cost. RMSNorm needs one reduction (∑x²) instead of two, and drops a term from the backward. Zhang & Sennrich measured 7–64% speedups on the norm op depending on architecture and framework. In a modern LLM the norm is a memory-bandwidth-bound op sandwiched between two GEMMs, so the real win is a fused kernel that keeps the tensor in registers, not the FLOP count.
• Parameters. γ only, no β. Which is D instead of 2D per norm — irrelevant for memory, but it removes a bias that can carry a constant offset into every downstream layer.
• Interpretability / mechanistic work. LayerNorm's centring step is a rank-reducing projection: it maps ℝ^D onto a (D−1)-dimensional subspace. If you are doing anything with the residual-stream basis, RMSNorm is the friendlier operator because it's a pure scaling.
• Folding. Because centring is a linear operation, it can often be folded into the preceding linear layer's weights by centring the weight columns. Recent work (Enjoy Your LayerNorm with the Efficiency of RMSNorm, 2025–26) formalizes when an LN in an arbitrary graph is "foldable" and reports 2–12% end-to-end inference speedups from converting LN→RMSNorm losslessly. Relevant if you're deploying a LayerNorm-era ViT next to an RMSNorm LLM.

And a fourth variant you will meet in code: Gemma's RMSNorm computes (1 + w) ⊙ x̂ rather than w ⊙ x̂, so the weight is initialized at zero and the layer starts as an identity-scale. Numerically identical family, different init and different weight-decay semantics. If you port Gemma weights into a LLaMA-shaped implementation and forget this, your model outputs garbage — a classic, and it is one line.

Beyond the big four there is a long tail, and a surprising amount of it is alive in code you use daily — AdaIN inside style models, adaLN-Zero inside every DiT, Weight Standardization inside BiT and NFNets, spectral norm inside GAN discriminators.

Choosing RMSNorm over LayerNorm is a small decision. Choosing where the norm sits relative to the residual addition is one of the largest architectural decisions in a transformer, and it is still being actively re-litigated in 2025–26.

Start with the two classics, written so the difference is unmissable:

Post-LN puts a normalization in the gradient's way at every single block. Gradient magnitude at layer ℓ decays roughly as O(1/√ℓ) going backwards, which is why the original transformer needed a learning-rate warmup and why deep Post-LN models are notoriously fragile. Pre-LN gives the gradient an uninterrupted identity path from the loss to the embedding. It trains without warmup, it survives depth, and it became the default.

But Pre-LN has a cost that took the field years to name properly. Because the residual sum is never renormalized, variance accumulates monotonically with depth:

and as Var[x_ℓ] grows, Norm(x_ℓ) becomes ever less sensitive to the branch's contribution, so the block's Jacobian drifts toward the identity. The deep layers stop doing anything. This is the Curse of Depth (§08), and it is the reason four different fixes shipped in 2025.

The eleven schemes, in the order they were invented

Post-LN — Vaswani et al., 2017

The original. Best final quality per parameter in several controlled comparisons, worst trainability. Needs warmup, careful init, and dies above ~20 layers without help. Still used deliberately: BERT-family encoders, and GLM.

Pre-LN — Baevski & Auli 2019; Xiong et al., 2020

The default for GPT-2 through Llama 3, Mistral, Qwen 2/3, Phi, DeepSeek. Clean identity path, no warmup needed, stable to hundreds of layers. Pays for it with unbounded residual-stream variance and the massive-activation phenomenon.

Sandwich-LN — Ding et al. (CogView), 2021

Add a second norm at the end of each residual branch: x ← x + Norm(F(Norm(x))). Invented for CogView, a text-to-image transformer, precisely because mixed-modality training blew up in fp16. It bounds each branch's contribution. GLM-130B tried it at 130B scale and reported it still failed to stabilize training — a useful negative result.

DeepNorm — Wang et al. (DeepNet), 2022

Post-LN with two scalars: upweight the residual and downweight the branch's initialization.

DeepNet trained a 1000-layer transformer with it. GLM-130B adopted DeepNorm-initialized Post-LN after Pre-LN, Post-LN and Sandwich-LN all failed at 130B — which is why the whole GLM/ChatGLM/CogVLM lineage is a Post-LN family in a Pre-LN world.

Swin-V2 res-post-norm — Liu et al., 2022

Move the norm to the output of each branch, still inside the residual: x ← x + Norm(F(x)). Liu et al. introduced it to stop activation amplitudes exploding when scaling Swin to 3B params. It is the single most consequential placement idea of the last few years, because two very different teams picked it up for the same reason (below).

Chameleon & OLMo 2 "reordered norm" — 2024

Both are the Swin-V2 scheme, adopted independently to survive training runs that were diverging. Chameleon needed it at 34B because SwiGLU's multiplicative structure made FFN output norms grow unboundedly under mixed-modal data; OLMo 2 needed it (together with QK-Norm) to tame gradient-norm spikes. OLMo 2 is explicit that neither change worked alone — reordering and QK-Norm only helped in combination.

Peri-LN — Kim et al., 2025

Norm both the input and the output of every sublayer: x ← x + Norm(F(Norm(x))). Yes — that is Sandwich-LN. The Peri-LN paper's contribution is not the shape but the analysis: it names the scheme that Gemma 2/3 and OLMo-family models had "silently adopted without much explanation", and shows it lands between Pre-LN's exponential variance growth and Post-LN's vanishing gradients, with steadier gradient flow up to 3.2B params. If you look at a Gemma 2/3 block you will find four RMSNorms: input_layernorm, post_attention_layernorm, pre_feedforward_layernorm, post_feedforward_layernorm. That is Peri-LN in HuggingFace's naming.

Mix-LN — Li et al., 2025

Post-LN for the early layers, Pre-LN for the deep ones. Motivated by the observation that Post-LN under-trains deep layers while Pre-LN under-trains them for a different reason; use each where it doesn't hurt.

HybridNorm — Zhuo et al., 2025 (NeurIPS)

Splits the decision within a block instead of across blocks: QKV-Norm inside attention, Post-Norm on the FFN. The attention sublayer gets Pre-Norm-like gradient flow (normalizing Q, K and V right before the attention computation), and the FFN gets Post-Norm's representational benefit. Reported to beat both Pre-Norm and Post-Norm and to edge out Mix-LN, on dense and MoE models. The HybridNorm* variant treats the first block specially with plain Pre-Norm.

LayerNorm Scaling — Sun et al., 2025 (NeurIPS)

The most surgical fix: keep Pre-LN, but multiply the output of the norm in layer ℓ by 1/√ℓ.

It caps the layerwise output-variance explosion (their LLaMA-130M measurements go from ~175 in the final layer down to a bounded value) and improves both pretraining loss and downstream SFT from 130M to 7B. Converges slightly slower early, then pulls ahead.

Depth-Scaled Sandwich-Norm — Pangu, 2025

Sandwich-Norm with depth-dependent scaling on the post-module norms. Huawei reports zero loss spikes across the entire pretraining trajectory of a 38B model, and a separate trick for absorbing the post-RMSNorm at inference time to recover the ~6% throughput it costs.

A finding that reframed a lot of folklore: in Pre-LN LLMs, the deepest layers barely contribute. Pruning them costs little. Removing them costs little. Perturbing them costs little. That is not a data problem — it's an arithmetic consequence of Pre-LN.

The mechanism, in three steps. First, residual-stream variance accumulates with depth because nothing renormalizes it. Second, the block's contribution is F(Norm(x)), whose magnitude does not grow with ‖x‖ — the norm sees to that. Third, therefore the ratio (branch contribution : residual) shrinks with depth, so ∂x_ℓ+1/∂x_ℓ → I. The layer becomes an identity map with a rounding error attached.

The same picture explains several things at once:

• Why deep layers prune well. Reported repeatedly for Llama, Mistral, Qwen and DeepSeek before anyone had a mechanism.
• Why removing early layers is catastrophic and removing late layers is not. Early layers operate on a low-variance stream where their contribution is large relative to the residual.
• Why "massive activations" appear. Very large-magnitude features in a few channels of a few tokens are one way the network can locally modulate the effective gain of a Pre-LN block, since Norm divides by the stream's own scale. Recent work frames them as adaptive gradient regulators for attention-sink tokens rather than as pathology.
• Why scaled init isn't enough. Scaling initialization by 1/√(2L) (GPT-2's trick) only sets the starting point. The variance explosion is a training-time phenomenon; by 2k steps the deep layers have grown their variance back. LayerNorm Scaling works because it is applied at every forward pass, not once.

Post-LN has the mirror-image problem: variance stays flat (the norm on the highway sees to it), but gradients attenuate through every block, so the early layers are the ones that under-train. Mix-LN's proposal — Post-LN early, Pre-LN late — is exactly a statement that the two pathologies have opposite depth profiles.

There are two softmaxes in a transformer. Everyone remembers the one over the vocabulary. The one over attention keys is the one that kills training runs.

Attention logits are qᵀk/√d. Nothing bounds them. As ‖q‖ and ‖k‖ grow during training — and in a Pre-LN model, they grow, because the residual stream grows — the logits grow quadratically, the softmax saturates, and attention entropy collapses to a one-hot. At that point the gradient through attention vanishes, the loss spikes, and in bf16 you are also flirting with the top of the representable range. Zhai et al. named this attention entropy collapse in the ViT-22B work; Dehghani et al. shipped the fix in the same paper.

Ordering detail that trips people up: QK-Norm goes after the q/k projections and before RoPE in most implementations (Gemma 3, OLMo 2). Normalizing before rotation is fine because RoPE is a rotation — it preserves norms — so the two commute in the only way that matters.

Who uses it, and what it cost them

• ViT-22B (2023) — where it was introduced, to fix divergence at 22B vision params.
• Chameleon (2024) — mixed-modal early-fusion. Without QK-Norm the 7B model diverged about 20% into an epoch. The paper's framing is worth reading: because softmax is shift-invariant, modalities "compete" for representation space in the shared logit range, and image tokens have a very different entropy profile from text. QK-Norm bounds the competition. It still wasn't enough at 34B, which needed the norm reordering as well.
• OLMo 2 / OLMoE (2024–25) — QK-Norm plus reordered norm. OLMoE measured QK-Norm costing nearly 10% throughput and shipped it anyway because it removed a large gradient-norm spike.
• Gemma 3 (2025) — replaced Gemma 2's attention logit soft-capping with QK-Norm, gaining both accuracy and speed (soft-capping is incompatible with efficient attention kernels; QK-Norm isn't).
• Qwen3, Kimi, most 2025+ frontier releases — near-universal now. Qwen3-Next uses a zero-centred RMSNorm variant inside its gated attention.

A persistent research thread argues that normalization layers are a crutch for bad initialization and bad residual scaling — and that if you fix those directly, you can delete the norms. The thread keeps producing results that work, and keeps failing to displace the default.

Fix the initialization instead

• Fixup (Zhang et al., 2019) — rescale the branch weights by L^{−1/(2m−2)}, zero-init the last layer of each branch, add scalar biases and a scalar multiplier. Trains 10,000-layer ResNets without any normalization.
• SkipInit / ReZero (2020) — the minimal version: put a learnable scalar α, initialized to 0, on each residual branch. x ← x + α·F(x). At init the network is exactly the identity, so signal propagation is perfect; α grows as training decides each layer is worth using.
• LayerScale (CaiT, 2021) — the same idea with a per-channel learnable vector initialized to ~1e-4 or 1e-5 instead of a scalar. Used with norms, not instead of them. Ubiquitous in modern vision transformers: DINOv2, EVA, InternViT, most ViT-G-scale encoders. If you're wondering why your ViT has a ls1 / gamma_1 parameter, that's LayerScale.
• Admin (2020) — profiling-based init for deep Post-LN transformers; stabilizes 100-layer Post-LN encoder-decoders.

Fix the whole network instead

NFNets (Brock et al., 2021) replace BatchNorm in a ResNet with three ingredients: Weight Standardization (normalize each conv filter's weights to zero mean, unit variance), explicit scalars that emulate BN's variance-preserving effect, and Adaptive Gradient Clipping — clip each parameter's gradient relative to ‖W‖ rather than to a global constant. The result matched or beat EfficientNet while training substantially faster. Flamingo's vision tower is one.

Replace the norm with a pointwise squashing function

Dynamic Tanh (Zhu, Chen, He, LeCun, Liu — CVPR 2025) is the most interesting recent entry, because its motivation is an observation rather than a theory: if you plot input-vs-output for a trained LayerNorm in a deep transformer, you get an S-curve. So replace the layer with an S-curve:

No reductions, no cross-element dependence, trivially parallel, and it matched or beat normalized baselines on ViT, ConvNeXt, MAE, DINO, DiT, LLaMA, wav2vec 2.0 and DNA models — usually without hyperparameter retuning. Two things it does not do: control how variance grows with depth (there's no per-token rescaling, so the Pre-LN variance story is unchanged), and it needed α init tuning for the largest LLMs.

The follow-ups are already in: Derf (2025) searches over pointwise functions and lands on erf(αx + s), reporting wins over LayerNorm, RMSNorm and DyT across vision, speech and DNA; and a 2026 line on bounded-tanh variants that explicitly re-add depth-dependent variance control, which is the part DyT left on the table.

Normalize everything, everywhere, always

The opposite extreme: nGPT (2024) constrains every embedding, every hidden state, and every weight matrix row to unit norm, so the entire model lives on a hypersphere. Residual updates become learned-rate spherical interpolations rather than additions. Reported 4–20× fewer steps to a given loss. The catch is that it's a whole-architecture commitment, not a drop-in.

Everything above is architecture. This section is the operational layer — the details that don't appear in papers but appear in your loss curve.

1 · Compute norms in fp32, always

Under bf16 autocast, ∑x² over 4096 elements with values around 10² will lose precision fast, and 1/√(·) amplifies the error. Every serious implementation upcasts to fp32 inside the norm and downcasts on the way out. If you write your own RMSNorm kernel and skip this, you will see a loss curve that is almost right and drifts.

2 · Where ε sits changes the answer

Different functions. Combined with ε values ranging from 1e-5 to 1e-12 across model families, this is a real source of conversion bugs. Check it when porting checkpoints.

3 · Exclude norm parameters (and embeddings) from weight decay

Decaying γ toward zero shrinks every activation the layer emits, which — per §01 — is a learning-rate change disguised as regularization. Standard practice is to build two parameter groups: everything with ndim ≥ 2 gets decay, biases and norm gains don't. OLMo 2 additionally excludes embeddings, having found that decaying them shrinks embedding norms enough to magnify gradients downstream.

4 · Gemma's (1 + w) convention

Covered in §05. Also affects decay: with w initialized at 0, weight decay pulls the layer toward identity scale rather than toward zero scale. Arguably the more sensible parameterization.

5 · The final norm before the LM head

Easy to forget, load-bearing. It's the last thing standing between an unbounded residual stream and a 100k-way softmax. It's also where LN-folding tricks pay off most, and where the "logit lens" family of interpretability methods does its work.

6 · Massive activations and attention sinks

In every Pre-LN LLM, a small number of channels on a small number of tokens (usually BOS) carry activations orders of magnitude above everything else, and those tokens absorb enormous attention mass. Recent work (2025–26) argues these are not a bug: massive activations act as gradient regulators, exploiting the 1/σ factor in RMSNorm's backward to attenuate gradient pressure created by attention sinks, and they simultaneously produce the "compression valley" seen in mid-layer representations. Interventions that suppress them without removing the sink (V-scale, gated-rescaling variants) improve quantization robustness. Practically: if you're quantizing, these outliers are your entire problem, and they are norm-mediated.

7 · Norm folding & FlashNorm

RMSNorm followed by a bias-free linear layer is scale-commutative, so the norm's γ can be merged into the next weight matrix and the division deferred past the GEMM. Free at inference. Better still: if you have QK-Norm, the pre-attention RMSNorm becomes mathematically redundant — norm → linear → norm collapses — and can be deleted outright with its weights folded into QKV. There are open feature requests on frontier model repos to do exactly this.

8 · Norms are the sequence-parallel sync points

Under tensor parallelism, LayerNorm/RMSNorm need the full hidden dimension, so they're the natural boundary for sequence-parallel regions: the all-gather before the norm and reduce-scatter after are what Megatron's sequence parallelism is built around. Under FSDP2 the norm parameters are tiny DTensors that are cheap to shard but generate a lot of small collectives — one reason people group them.

9 · Fuse the kernel, and think about recompute

RMSNorm is bandwidth-bound: it reads the tensor, reduces, reads again, writes. Fused implementations (Apex, Triton, Liger) do it in one pass and can fuse the residual add too. Under activation checkpointing, norms are the canonical thing to recompute rather than store — cheap FLOPs, large activation.

10 · Zero-centred gains, and QK-Norm's own weights

QK-Norm adds a norm per head over the head dimension, so its parameters are d_head-sized, not d_model-sized. Whether they're shared across heads is a real architectural choice with different names in different codebases; per-head is now more common. Qwen3-Next uses a zero-centred RMSNorm variant here.

11 · Norm growth is your best early-warning signal

Chameleon's most transferable finding: monitoring the L2 norm of the final transformer layer's output predicted loss divergence 20–30% of training in advance. The loss curve looked fine; the output norm was already climbing. Log it. Also log per-layer gradient norms — OLMo 2 diagnosed its interventions on gradient-norm spikiness, not on loss.

12 · Data can cause a "normalization" problem

OLMo 2 traced a chunk of their loss spikes to training batches containing long repeated n-grams. No architectural fix addresses that; they filtered the data. Worth remembering before you spend a week moving norms around.

Bolt a Pre-LN vision encoder onto a Pre-LN language model and you have joined two residual streams that grew their variance independently, for different numbers of layers, under different objectives. Nobody normalized between them. This is, measurably, a problem.

The numbers nobody puts in the tech report

A 2025 study measured the L2 norm of tokens on both sides of the interface, over 1000 samples from MMBench / POPE / MM-Star:

Vision encoder outputs sit at L2 norms of roughly 29 (CLIP ViT-L/14), 72 (SigLIP-SO400M), 59 (SigLIP 2) and 72 (MoonViT). Text embeddings in Qwen2.5-7B, Qwen3-8B and Llama-3.2-3B sit at 0.80, 1.38 and 1.09. That is a 30–90× gap going into a shared Pre-LN backbone.

Does the projector fix it? Partly, and inconsistently. Measured after the connector: LLaVA-1.5 ≈ 40, Qwen2.5-VL ≈ 57, Kimi-VL ≈ 4.8, GLM-4.1V ≈ 4.6 — against text at ≈ 0.8–1.1. Two orders of magnitude of spread between VLMs, on a design axis that essentially none of their technical reports discuss.

Why a norm gap becomes a fusion problem

In a Pre-LN block the update magnitude ‖Δh‖ = ‖F(Norm(h))‖ is decoupled from ‖h‖ — the norm strips the scale before the branch sees it. So every token gets an update of roughly the same size, but tokens have wildly different lengths. A fixed-length update rotates a short vector a lot and a long vector barely at all:

High-norm visual tokens therefore have representational inertia: they rotate through semantic space more slowly than text tokens, layer after layer. The two modalities diverge geometrically instead of converging, the dot products that attention depends on are capped below what a balanced model could reach, and the signal-to-noise ratio of text→image attention collapses. You see it as diffuse, unfocused attention maps and as the model quietly falling back on its language prior.

Two corroborating observations from the same study, both of which are the kind of thing you'd only find by looking:

• The model tries to compensate through the embedding table. In Qwen2-VL, the <|vision_start|> and <|vision_end|> embeddings grow to L2 norms of 2.46 and 2.33 — while the largest norm of any token in the base Qwen2-7B is 1.03. The vision boundary tokens become the two biggest vectors in the vocabulary. Qwen2.5-VL shows the same pattern and a systemic inflation of the whole embedding matrix (mean norm 0.80 → 0.90 after VL training).
• Models with a smaller norm gap show less update asymmetry. Kimi-VL and GLM-4.5V — whose connectors compress hard to ≈ 4.6–4.8 — have visibly more synchronized per-layer update rates than LLaVA-1.5 and Qwen2.5-VL.

The proposed fix is one LayerNorm after the projector, with its gain initialized to T/√D where T is the mean text-embedding norm. There's a wrinkle worth knowing: that init is tiny (~0.01), and since the gradient to the vision encoder gets multiplied by the gain, naive initialization vanishes the gradient and detaches the encoder from supervision. They fix it with a backward hook that rescales by 1/ḡ, decoupling forward scale from backward scale. On LLaVA-1.5 + SigLIP-SO400M the intervention improved multimodal benchmarks and text-only MMLU (+8 points with Llama-3.2-3B), which is the tell that it was fixing an architectural imbalance rather than trading modalities off.

Nuance 2 — the encoder and the LLM disagree about what a norm is

Historically, vision encoders are LayerNorm and language models are RMSNorm. That's a mixed-operator model: LayerNorm's centring in the tower, none in the backbone. Qwen was the first big family to fix it deliberately — Qwen2.5-VL rebuilt its ViT with RMSNorm and SwiGLU specifically to match the Qwen2.5 LLM structure. Pixtral's vision encoder likewise uses RMSNorm to match Mistral. It is not obvious this matters for quality; it definitely matters for kernel reuse, quantization tooling, and the number of code paths you maintain.

Nuance 3 — mixed-modal early fusion is where norms actually break

The LLaVA-style architecture is forgiving because the LLM is frozen or lightly tuned and visual tokens are a minority of the sequence. Early-fusion token-based models are not. Chameleon is the canonical case study, and its diagnosis is precise: softmax is shift-invariant, so image and text tokens compete for the shared logit range; their differing entropy profiles drive slow norm growth in the mid-to-late training; and eventually the norms leave bf16's usable range. The fix took three ingredients — QK-Norm, then dropout after attention and FFN (7B), then Swin-style norm reordering (34B, where dropout alone failed).

The generalizable lesson: adding a modality changes the entropy of your attention distributions, and normalization is what absorbs that change. If you are moving from a late-fusion adapter to native mixed-modal training, expect to re-tune your norm placement, not just your data mix.

Nuance 4 — the norm you forget is the one on the pixels

Input preprocessing is a normalization layer with hardcoded, non-learned statistics, and it is the single most common source of silent VLM degradation. CLIP uses (0.481, 0.457, 0.408) / (0.268, 0.261, 0.276). SigLIP uses (0.5, 0.5, 0.5) for both. ImageNet uses a third set. OpenCLIP defaults differ from OpenAI CLIP defaults. Swapping an encoder without swapping the constants produces a model that works — just several points worse, with no error anywhere. Check image_processor_config.json against the encoder you actually loaded.

Nuance 5 — frozen BatchNorm in hybrid towers

Anything with a CNN in the vision path — ConvNeXt in Mini-Gemini's high-res branch, SAM's ViT-Det neck, EVA's conv stem, ResNet-based detectors distilled into VLM pipelines — carries BatchNorm or a BN-descendant. Under VLM training your batch is a handful of images per device with wildly varying resolution and aspect ratio. Running statistics computed on ImageNet-like batches are simply wrong for that distribution. Freeze them, or convert to GroupNorm before you fine-tune.

Nuance 6 — resolution, packing, and per-token statistics

Native-resolution encoders (NaViT-style packing, Qwen2-VL, MoonViT, Kimi-VL) put images of different sizes in one packed sequence. LayerNorm and RMSNorm are per-token, so this is fine — which is precisely why native-resolution ViTs use them and could not use BatchNorm. But two things do change: token counts per image vary, so anything downstream that averages over tokens (attention pooling connectors, mean-pool heads) sees a variable-length reduction; and window-attention layers mixed with full-attention layers (Qwen2.5-VL's 4-full / rest-windowed pattern) produce different activation scales per layer type. If you see per-layer norm statistics that jump periodically, that's usually the attention pattern, not a bug.

Nuance 7 — where the connector's norms actually go

• Linear / MLP projector (LLaVA, most models) — usually no normalization at all. Cheapest, and the one with the worst measured norm alignment.
• Perceiver resampler / Q-Former (Flamingo, Idefics, MiniCPM-V, Qwen-VL) — a small transformer, so it has its own LayerNorms, plus typically a LayerNorm on the learned queries and on the cross-attention keys/values. Output norm is whatever that stack lands on.
• Attentional pooling (Molmo, CoCa-style) — a single cross-attention block; same story, plus the pooling reduces a variable number of tokens.
• Gated cross-attention (Flamingo, OpenFlamingo, Idefics 1) — LayerNorm plus a tanh gate initialized to zero, so the frozen LM is untouched at initialization and the visual pathway fades in. This is LayerScale/ReZero applied to the modality interface, and it is the reason Flamingo could bolt vision onto a frozen Chinchilla without destroying it.
• Modality-expert FFN/QKV (CogVLM's visual expert, MoE-style routing) — separate weights per modality inside the block. The norms are typically still shared, which means the two modalities' statistics are pooled by the very layer that was supposed to separate them.

Case study — the Gemma / PaliGemma line

One family, four generations, and every normalization decision in this guide shows up at least once. It is the cleanest worked example available, so it's worth walking end to end.

• PaliGemma (2024) — SigLIP-So400m/14 into a single linear layer into Pre-LN Gemma-2B. No normalization anywhere in the connector. On the measurements in Fig 10 this is the worst case in the matrix: SigLIP-SO400M produces the highest-norm visual tokens of any tower listed, and Gemma's embedding table sits three orders of magnitude below. It trains anyway, because the tower is unfrozen and the whole stack sees a very large multi-stage curriculum — the linear map is free to absorb the scale. The lesson is not “projector norms don't matter”; it's that end-to-end training at scale can buy you out of an unaligned interface, and nothing else can.
• PaliGemma 2 (2024) — identical tower, identical bare projector, but the decoder is Gemma 2, which means the residual stream is peri-LN. That single change alters the failure mode: a post-sublayer RMSNorm at the end of every branch re-bounds each contribution, so an over-scaled visual token no longer compounds down the depth of the model. If you want an argument for sandwich placement in mixed-modal stacks that isn't hand-waving, this is it.
• Gemma 3 (2025) — native multimodality rather than an adaptation stage, average-pooling the tower down to a fixed 256 tokens, and QK-Norm replacing Gemma 2's attention logit soft-capping. That swap is the single best illustration in the guide of §09's argument: soft-capping and QK-Norm solve the same problem — attention logits growing until the softmax saturates — but soft-capping is a pointwise tanh on the score matrix, which forces you out of fused attention kernels. Normalizing q and k before the score is computed leaves the kernel intact. Better accuracy and faster, from moving the fix upstream.
• Gemma 3n (2025) — the on-device variant keeps peri-LN and QK-Norm, then adds RMSNorms along every side path that MatFormer, AltUp, LAuReL and the per-layer embeddings introduce. It has more distinct normalization sites than anything else in the matrix, which is what happens when your architecture stops being a single residual stream.

Seventy models, three columns that matter: the operator, the placement, and whether there's a norm inside attention. VLM rows also carry the vision tower and the connector, because in a VLM those are three different normalization regimes stapled together.

Filter by family or search. Entries marked unclear are ones where the technical report doesn't say and I haven't verified against weights — treat those as prompts to go look, not as facts.

Reading the matrix

• The Pre-LN + RMSNorm monoculture is real but cracking. Llama, Mistral, Qwen 2/3, Phi, DeepSeek, InternLM and everything built on them are one architecture. The deviations — GLM's DeepNorm Post-LN, Gemma's Peri-LN, OLMo 2's reordered norm, Chameleon's Swin reordering — were all adopted after a specific training run failed.
• QK-Norm went from exotic to default in about two years. ViT-22B (2023) → Chameleon and OLMo 2 (2024) → Gemma 3, Qwen3 and most 2025 frontier releases. If you're designing now, it's the cheapest stability insurance available.
• Vision towers lag their backbones by a generation. Most VLMs in 2024–25 still pair a LayerNorm CLIP/SigLIP tower with an RMSNorm LLM. Qwen2.5-VL and Pixtral are the notable exceptions that rebuilt the tower to match.
• The GLM/CogVLM lineage is a genuine outlier — Post-LN with DeepNorm, descending from GLM-130B's finding that Pre-LN, Post-LN and Sandwich-LN all failed at 130B. Worth studying if you're above 100B.
• Flamingo remains the strangest entry — an NFNet vision tower with no normalization layers whatsoever, feeding a Perceiver Resampler, feeding zero-initialized tanh-gated cross-attention into a frozen LM. Three completely different normalization philosophies in one model.

Seventy-five papers, grouped by the question they answer rather than by date. Every title links to the paper.

Each entry also carries its arXiv identifier as a second link that runs a title search, so if an identifier is ever stale or wrong the paper is still one click away. Entries marked search were not published on arXiv.