A field guide · Understanding LLMs

What Changed Inside the Transformer?

The 2017 Transformer and a 2026 frontier model share a skeleton. Almost every organ has been replaced. This is a part-by-part inventory of what changed, and the pressure that forced each change.

TransformersArchitectureKV Cache
Anirudh SrinivasanPublished Last modified 52 min read

Same skeleton, new organs

Component2017 defaultModern defaultQuality gain?Pressure that drove it
PositionAdded absolute embeddingRoPE inside attentionSmallLength extrapolation and relative geometry
NormalizationPost-LayerNormPre-RMSNormSmallStable deep, low-precision training
FFNReLU, two matricesSwiGLU, three matricesModerateBetter parameter use
KV headsOne pair per query headGrouped or latent KVUsually neutralDecode memory bandwidth
ContextDense attentionSliding / hybrid layersTask-dependentKV-cache growth
CapacityOne dense FFNSparse experts at scaleScaling gainMore parameters without equal FLOPs

The trend. The block changed less to improve next-token prediction than to survive scale: numerical range during training, memory traffic during decoding, and the cost of activating every parameter on every token.

Open the modeling file for Llama 3, Qwen3, DeepSeek-V3, or Olmo 3 and you will find something that would be legible to someone who had only read Attention Is All You Need. Embed tokens. Loop over blocks. Each block: attention, residual add, feed-forward network, residual add. Project to vocabulary. Softmax. That is still the whole program.

And yet almost nothing inside those four steps is the same object it was in 2017. The normalization layer computes different statistics and sits in a different place. Position information is no longer added to the input; it is applied inside attention as a rotation. The feed-forward network has three weight matrices instead of two, and in the largest models there are two hundred and fifty-six of them per layer with only nine running per token. Attention no longer gives each head its own keys and values. Three quarters of the layers cannot see past the last four thousand tokens.

The interesting thing is that almost none of these changes were motivated by better language modeling. Sweep the ablations and the accuracy deltas are mostly within noise. What drove them was a different set of pressures: memory bandwidth during decoding, numerical stability at low precision and large scale, the cost of a cross-feature reduction on a GPU, the fact that a 128k-token KV cache does not fit next to the weights, and — above everything else — the desire to add parameters without adding FLOPs.

This article is an inventory. For each part of the block, I want to do three things: state what the 2017 version did, derive why it stopped being the right choice, and show what replaced it in models you can download today. Where the community has genuinely not settled a question, I will say so rather than manufacturing a clean story.

The animation below is the whole argument in fourteen steps. Each step applies exactly one change to the 2017 decoder block and holds everything else fixed. The rest of the article expands one step at a time.

Figure 1 · The morph

One change at a time, 2017 → 2026

Frame 01 / 14

Autoplay pauses while you step manually and resumes about twelve seconds after your last interaction. Arrow keys step frames when the figure has focus.

Two honest caveats before we start. First, architecture is probably not where most of the last eight years of capability came from. Data scale, data quality, and post-training moved the benchmarks far more than any of these components did, and the labs that publish ablations tend to report architecture deltas of a fraction of a point. Second, these changes are not independent. QK-norm exists partly because RMSNorm removed a stabilizer. Sliding-window attention is affordable partly because RoPE handles relative position gracefully. I will flag the couplings as they come up.

Companion pieceIf you want the longer arc — perceptrons through CNNs, RNNs, Transformers, and state-space models — Evolution of ML Architectures covers the sixty-seven years before this article starts.

The residual stream is the object

Before touching any component, it helps to fix the right mental picture of what a Transformer block is, because every later design decision is easier to read against it.

A decoder-only Transformer maintains one tensor of shape \((B, S, d)\) from the embedding layer to the final norm. Call it the residual stream. Every sublayer reads the stream, computes something, and adds the result back. Nothing ever overwrites it:

$$x_{\ell+1} = x_\ell + \mathrm{Attn}_\ell\!\big(\mathrm{Norm}(x_\ell)\big) + \mathrm{FFN}_\ell\!\big(\mathrm{Norm}(x_\ell + \mathrm{Attn}_\ell(\cdots))\big)$$

Pre-norm form. Each sublayer contributes an additive delta; the stream itself is never transformed in place.

Three consequences follow immediately, and they explain a surprising fraction of modern design.

The stream is a shared bus with a fixed width. Every layer writes into the same \(d\)-dimensional space, so features have to be arranged in a way that later layers can find. This is why \(d\) is expensive to grow — attention parameters scale as \(d^2\), FFN parameters scale as \(d \cdot d_\text{ff}\), and the KV cache scales linearly in \(d\) too. Almost every efficiency trick of the last three years is a way to buy capacity without growing \(d\).

The identity path means gradients reach layer 1 unattenuated. The Jacobian of the residual add is \(I + \partial(\text{sublayer})\). Stack \(L\) of them and you get a product whose dominant term is the identity. That is the whole reason a 94-layer model trains at all. It is also why where you put the normalization matters so much: put it on the identity path and you break it, put it on the branch and you preserve it. Section 4 is entirely about this.

Variance accumulates. If each sublayer writes a roughly independent delta of similar scale, then \(\|x_\ell\|^2 \approx \|x_0\|^2 + \ell\sigma^2\), so \(\|x_\ell\|\) grows like \(\sqrt{\ell}\). Real models show exactly this: the residual norm climbs monotonically with depth, often by an order of magnitude. That growth is benign as long as every consumer of the stream normalizes first — and catastrophic if some consumer does not.

The figure below traces one token's stream through a 24-layer model, showing the deltas being written, the norm growing, and where a normalization layer intervenes under three different placement conventions.

Figure 2 · The residual stream

Deltas accumulate; norms decide who sees what

Frame 01 / 13

Keep this picture. When we get to multi-head latent attention, sliding windows, and mixture-of-experts routing, each one is a statement about what a layer is allowed to read from the bus and how much it is allowed to write.

Where the position went

Attention is permutation-equivariant. Shuffle the tokens in the input and the attention output shuffles identically — the mechanism has no idea which token came first. Position has to be injected somewhere, and the history of how we inject it is one of the cleanest stories in the field.

Three generations of answer

2017: add a sinusoid. The original Transformer added a fixed vector \(p_m\) to the token embedding, with \(p_{m,2i} = \sin(m/10000^{2i/d})\) and cosine in the odd slots. Fixed, not learned, and chosen so that \(p_{m+k}\) is a linear function of \(p_m\) — the paper hoped this would make relative offsets easy to learn.

2018–2020: learn it. GPT-2 and GPT-3 replaced the sinusoid with a learned lookup table \(P \in \mathbb{R}^{S_{\max} \times d}\), added to the embedding. Simple, works, and has a hard failure mode: row \(m\) for \(m \ge S_{\max}\) does not exist. GPT-2 could not process a 1025th token at all. Extending context meant training new rows from scratch.

Both share a deeper problem. Position enters once, at the input, as an additive perturbation of the content vector. By layer 40 the model has to have preserved that perturbation through forty rounds of mixing in order to still know where it is. And the thing attention actually cares about is not absolute position — it is the offset between the query and the key.

The requirement, stated properly

Here is the design question rotary embeddings answer. We want a function \(f\) that takes a vector and a position, such that the attention logit between query at position \(m\) and key at position \(n\) depends only on the two vectors and their difference:

$$\big\langle f(q, m),\; f(k, n) \big\rangle \;=\; g\big(q,\, k,\, m-n\big)$$

Relative position for free, in the inner product, with no extra terms in the attention score.

In two dimensions there is an obvious candidate: rotation. Let \(R_\theta\) be the 2×2 rotation matrix by angle \(\theta\), and set \(f(q, m) = R_{m\theta}q\). Then, because rotations are orthogonal and \(R_a^\top R_b = R_{b-a}\):

$$\langle R_{m\theta}q,\; R_{n\theta}k\rangle \;=\; q^\top R_{m\theta}^\top R_{n\theta} k \;=\; q^\top R_{(n-m)\theta}\, k$$

The absolute positions cancel exactly. Only \(n-m\) survives.

That is the whole idea. RoPE splits the \(d\)-dimensional head into \(d/2\) coordinate pairs and rotates pair \(i\) by \(m\theta_i\), with the frequencies geometrically spaced:

$$\theta_i \;=\; b^{-2i/d}, \qquad i = 0, 1, \dots, \tfrac{d}{2}-1$$

\(b\) is the RoPE base. Pair 0 rotates once per token; the last pair completes a cycle every \(2\pi b\) tokens.

The geometric spacing gives the model a positional "clock" with hands at many speeds — fast pairs resolve nearby offsets precisely, slow pairs carry coarse long-range information. And critically, the rotation is applied to \(q\) and \(k\) inside every attention layer, not once at the input. Position is re-asserted at layer 40 rather than remembered from layer 0.

Figure 3 · Rotary position embeddings

Why a rotation is the right shape of answer

Frame 01 / 12

The base is a context-length dial

The slowest RoPE frequency has wavelength \(2\pi b\) tokens. If \(b = 10{,}000\), positions further apart than roughly 63,000 tokens start aliasing — the model has literally never seen a full period of its slowest dial during training, and offsets beyond training length map to angles it has no calibration for. This is why RoPE models degrade sharply past their training context rather than gracefully.

The field's response has been to turn the dial up and to rescale. Llama 2 used \(b = 10^4\). Llama 3 uses \(5\times10^5\). Qwen3 uses \(10^6\). Gemma 3 goes further and uses two bases: \(10^4\) for its local sliding-window layers, where only short offsets ever occur, and \(10^6\) for the sparse global layers that need to span the full context. YaRN and NTK-aware scaling go a step beyond retuning the base: they interpolate the low-frequency dials while leaving high-frequency ones alone, on the argument that fast dials already generalize and slow ones do not.

Where this is going

The most interesting recent move is removing position encoding. SmolLM3 and Arcee's Trinity Large use NoPE — no positional embedding at all — in their global attention layers, keeping RoPE only in the local ones. The causal mask alone already breaks permutation symmetry: token \(m\) sees \(m\) predecessors and token \(n\) sees \(n\), so the model can infer ordering from the attention pattern itself. Dropping RoPE from the global layers removes exactly the frequencies that fail to extrapolate.

Deep diveSinusoidal, learned, relative, ALiBi, RoPE, NoPE, 2D and M-RoPE, with animated derivations for each — How Models Know Where They Are.

Fewer statistics, more placements

Normalization changed along two independent axes: what it computes, and where it sits. The two are often conflated. They shouldn't be.

Axis one: LayerNorm → RMSNorm

LayerNorm standardizes across the feature dimension, then applies a learned scale and shift:

$$\mathrm{LN}(x) = \gamma \odot \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta, \qquad \mu = \tfrac{1}{d}\textstyle\sum_j x_j,\quad \sigma^2 = \tfrac{1}{d}\textstyle\sum_j (x_j-\mu)^2$$

RMSNorm drops the mean subtraction and the shift:

$$\mathrm{RMS}(x) = \gamma \odot \frac{x}{\sqrt{\tfrac{1}{d}\sum_j x_j^2 + \epsilon}}$$

One reduction instead of two. No \(\beta\). Scale control without re-centering.

Why does dropping the mean cost nothing? Because the thing being normalized is immediately consumed by a linear layer whose weights can absorb any constant offset, and because the residual stream's mean is not a meaningful quantity to begin with — it is an artifact of the embedding initialization, not a signal. The RMSNorm paper's claim, which has held up, is that re-scaling is the part that stabilizes optimization and re-centering is decoration.

The practical argument is sharper. A cross-feature reduction on a GPU is a warp-level or block-level sum: it forces a synchronization point and, under tensor parallelism where \(d\) is sharded across devices, an actual collective. LayerNorm needs two such reductions (mean, then variance about the mean); RMSNorm needs one. On a normalization layer that runs twice per block across 60–90 blocks, at every token, in a memory-bound regime, halving the reductions is a real throughput number rather than a rounding error.

Axis two: where it sits

The 2017 paper put normalization after the residual add: \(x_{\ell+1} = \mathrm{LN}(x_\ell + F(x_\ell))\). This puts a normalization layer directly on the identity path. Backpropagating through \(L\) of them multiplies \(L\) LayerNorm Jacobians together, and the clean identity term is gone. Post-LN Transformers famously require a learning-rate warmup to train at all, and the deeper they get the more fragile they become.

Pre-norm — \(x_{\ell+1} = x_\ell + F(\mathrm{Norm}(x_\ell))\) — moves the normalization onto the branch. GPT-2 already did this in 2019 and essentially every model since has followed. The identity path is now literally the identity, warmup requirements relax, and depth becomes cheap.

The cost is the variance growth from Section 2: with nothing constraining branch outputs, \(\|x_\ell\|\) grows unchecked and the effective contribution of deep layers shrinks relative to the accumulated stream. Two families of fix are in production:

Arcee's Trinity Large adds a third twist worth noting: the gain of the second RMSNorm in each block is initialized to roughly \(1/\sqrt{L}\), so early in training every branch writes a small delta and the model has to earn its way up to a larger residual scale. That is a depth-scaled initialization dressed up as a norm parameter.

QK-norm: the stabilizer nobody expected to need

Attention logits are \(q^\top k/\sqrt{d_h}\). Nothing bounds \(\|q\|\) or \(\|k\|\). If a head learns to increase both, logits grow, softmax saturates, gradients through it vanish, and in bf16 a large logit is also a precision problem. In practice this shows up as loss spikes.

QK-norm applies RMSNorm to \(q\) and \(k\) per head, immediately before the dot product. Bounding both vectors' scale caps the logit magnitude regardless of what the projections learn. It was popularized by the ViT-22B work and is now near-universal: Qwen3, Olmo 3, Gemma 3, and Trinity all use it.

Near-universal, not universal. Cohere's Tiny Aya dropped QK-norm, with the team's stated reason being that it can interact badly with long-context behaviour. That is a genuine open trade-off between training stability and inference behaviour, and I have not seen a clean public ablation resolving it.

Figure 4 · Normalization placement

The identity path is the whole game

Frame 01 / 11
Deep diveBatchNorm through Peri-LN, why each one was proposed, and what the geometry actually does to a vector — The Geometry of Normalization.

The FFN learned to gate

Two thirds of the parameters in a dense Transformer live in the feed-forward network, so a change here is a change to most of the model.

The 2017 FFN is two matrices with a pointwise nonlinearity between them, and an expansion factor of four:

$$\mathrm{FFN}(x) = W_2\,\phi(W_1 x), \qquad W_1 \in \mathbb{R}^{4d \times d},\; W_2 \in \mathbb{R}^{d \times 4d}$$

\(\phi = \mathrm{ReLU}\) in 2017, GELU by GPT-2. Parameter count: \(8d^2\).

The modern version is a gated linear unit with three matrices:

$$\mathrm{SwiGLU}(x) = W_2\Big(\underbrace{\mathrm{SiLU}(W_g x)}_{\text{gate}} \;\odot\; \underbrace{W_u x}_{\text{value}}\Big), \qquad \mathrm{SiLU}(z) = z\,\sigma(z)$$

Three matrices of shape \(d \times d_\text{ff}\). Parameter count: \(3\,d\,d_\text{ff}\).

Two changes, not one

It is worth separating them. The first is the activation: GELU \(\to\) SiLU/Swish. GELU requires the Gaussian error function, computed via polynomial approximation; SiLU needs one sigmoid. The curves are nearly identical and every honest ablation I have seen puts the modeling difference inside the noise band. This is a cost change, not a quality change. Google's Gemma line still uses GELU — in its gated form, GeGLU — which tells you how little rides on it.

The second change is the real one: the gate. The output is now a product of two learned projections of the same input. That multiplicative interaction is something a two-matrix MLP has to approximate with depth and width; here it is a primitive. The model can learn "route this feature through only when that other feature is present" in one layer, cheaply, per channel.

The 8/3 accounting

Adding a third matrix looks like it adds parameters. It doesn't, because the convention is to shrink \(d_\text{ff}\) to compensate. Setting \(3\,d\,d_\text{ff} = 8d^2\) gives:

$$d_\text{ff} \;=\; \tfrac{8}{3}\,d \;\approx\; 2.667\,d$$

Then rounded up to a hardware-friendly multiple, usually 128 or 256.

Llama 2 7B is the textbook case: \(d = 4096\), \(\tfrac{8}{3}d = 10{,}923\), shipped as 11,008 — the next multiple of 256. Llama 3 8B breaks the rule deliberately and uses 14,336 (\(3.5d\)), spending real extra parameters on the FFN because the team judged it worth the compute. So treat 8/3 as the parity convention, not a law.

Figure 5 · Feed-forward, gated

Three matrices, same budget, one extra multiply

Frame 01 / 09

A caution on the standard story

The usual claim is that GLU variants "perform better." The evidence for that is largely one 2020 paper of small-scale ablations plus the fact that everyone adopted it. Noam Shazeer's own conclusion in that paper was, essentially, that the architectures work and he offers no explanation for why — "divine benevolence" was the phrase. Nine years later there is still no clean mechanistic account of the gain. Adopt it because it is the standard and it is cheap, not because the mechanism is settled.

Deep diveSigmoid through squared ReLU, with derivations of each curve and a map of what production models actually ship — Why Neural Networks Need Nonlinearity.

The KV cache became the constraint

This is the section that explains the most changes, so it is worth being precise.

During autoregressive decoding you generate one token at a time. Every new query must attend to all previous keys and values. Recomputing them would be quadratic in the number of generated tokens, so you cache them. The cache is:

$$\text{bytes}_\text{KV} \;=\; 2 \times L \times n_\text{kv} \times d_h \times s \times b \times \text{bytes/elt}$$

The 2 is K and V. \(L\) layers, \(n_\text{kv}\) key/value heads, head dimension \(d_h\), sequence length \(s\), batch \(b\).

Put Llama-3-8B numbers in: \(L{=}32\), \(d_h{=}128\), bf16. With full multi-head attention (\(n_\text{kv} = 32\)) that is 512 KiB per token, so a single 128k-token sequence needs 64 GiB of KV cache — for a model whose weights are 16 GiB. One user would not fit on an H100.

The bandwidth argument, which matters more

Memory capacity is the obvious problem. The subtler one is bandwidth. At each decode step, attention in layer \(\ell\) must read the entire KV cache for that layer and does roughly \(4 n_q d_h s\) FLOPs against it. The arithmetic intensity is:

$$I \;=\; \frac{4\,n_q d_h s}{2 \cdot 2\, n_\text{kv} d_h s} \;=\; \frac{n_q}{n_\text{kv}} \quad\text{FLOPs per byte (bf16)}$$

Independent of \(s\), \(d_h\), and batch. It is purely the query-to-KV head ratio.

An H100 SXM does roughly 989 TFLOP/s of bf16 against 3.35 TB/s of HBM, so its ridge point is near 295 FLOPs per byte. Full multi-head attention gives \(I = 1\). Decode attention is therefore memory-bound by nearly three orders of magnitude: the tensor cores are idle almost the entire time, waiting on the KV cache to stream in.

That single number explains the entire MHA \(\to\) MQA \(\to\) GQA \(\to\) MLA progression. You cannot make the attention math cheaper — the FLOPs are already negligible. You can only make it read fewer bytes.

The progression

Multi-head latent attention, derived

MLA observes that keys and values are both linear functions of the same hidden state, and that a full-rank \(K\) and \(V\) per head is more information than the cache needs to carry. So compress once, decompress per head:

$$c_t \;=\; W^{DKV} h_t \in \mathbb{R}^{r}, \qquad k_t^{(i)} = W^{UK}_i c_t, \qquad v_t^{(i)} = W^{UV}_i c_t$$

DeepSeek-V3: \(r = 512\), against \(d = 7168\) and 128 heads of dimension 128.

Cache only \(c_t\). At decode time the up-projections never need to be applied at all, because they can be absorbed into the projections on the other side of each product:

$$q^\top k^{(i)} = q^\top W^{UK}_i c = \big(W^{UK\top}_i q\big)^\top c, \qquad \big(\text{softmax} \cdot v\big)W^O = \big(\text{softmax}\cdot c\big) W^{UV}W^O$$

Fold \(W^{UK}\) into the query projection and \(W^{UV}\) into the output projection. \(K\) and \(V\) are never materialized.

There is one obstruction, and it is the interesting part. RoPE applies a position-dependent rotation to \(k\). If you rotate after the up-projection, \(R_n W^{UK} \ne W^{UK} R_n\), so absorption breaks — you would need a different absorbed matrix per position. DeepSeek's fix is a decoupled RoPE key: a small extra 64-dimensional key, shared across all heads, that carries the rotation and is cached directly alongside \(c_t\). Everything else stays position-free and absorbable.

The result, for DeepSeek-V3's 61 layers:

$$\underbrace{61 \times (512 + 64) \times 2}_{\text{MLA}} = 70{,}272~\text{B/token} \;=\; 68.6~\text{KiB} \qquad\text{vs.}\qquad \underbrace{61 \times 128 \times (192{+}128) \times 2}_{\text{equivalent MHA}} = 4.77~\text{MiB}$$

A 71× reduction. At 128k context: 8.6 GiB instead of 610 GiB.

MLA is harder to implement and harder to shard than GQA, which is why it took two years to spread. It has now spread: GLM-5, Kimi K2.5, Ling 2.5, and Sarvam 105B all adopted it in the last year, while their smaller siblings stayed on GQA.

Figure 6 · KV cache

What each attention variant asks you to store

Frame 01 / 11
Systems companionWhy prefill and decode are two different machines, and what the KV cache does to batching, latency, and throughput — The Mechanics of LLM Inference.

The mask grew holes

GQA and MLA shrink the cache by a constant factor. Neither touches the fact that it still grows linearly in \(s\), or that attention FLOPs grow as \(s^2\). At 65k or 256k context, a constant factor is not enough.

Sliding-window attention is the blunt fix: restrict each query to the last \(w\) keys. Per-layer attention cost drops from \(O(s^2)\) to \(O(sw)\), and — the part that actually matters — the KV cache for that layer is capped at \(\min(s, w)\) tokens instead of \(s\). The cache stops growing.

The obvious objection is that the model can no longer see far. The answer is that it can, through depth. A stack of \(L\) local layers with window \(w\) has a receptive field of \(L \cdot w\) tokens, exactly like a stack of convolutions. Information travels, it just takes hops.

In practice nobody relies on hops alone. Every production design interleaves a few full-attention layers so that long-range information has a direct path:

ModelLocal : globalWindowNote
GPT-3 (2020)1 : 1banded"alternating dense and locally banded sparse", per the paper
Mistral 7B (2023)all local4096no interleaved global layers
Gemma 2 (2024)1 : 14096
Gemma 3 (2025)5 : 11024separate RoPE base per layer type
gpt-oss (2025)1 : 1128unusually small window
Olmo 3 (2025)3 : 14096full attention in the final layer
Trinity Large (2026)3 : 14096NoPE in the global layers

The spread of window sizes — 128 to 4096 — and ratios — 1:1 to 5:1 — is a good indicator that nobody has found the optimum. Gemma's own ablations report minimal quality impact across a wide range, which is consistent with the parameter simply not mattering much once you have some global path.

What it buys, in bytes

Take Olmo 3 32B at its full 65,536-token context: 64 layers, 8 KV heads, \(d_h = 128\), bf16. Per layer per token the cache is \(2 \times 8 \times 128 \times 2 = 4096\) bytes.

$$\text{full} = 64 \times 65536 \times 4096 = 16~\text{GiB}$$ $$\text{3:1 SWA} = \underbrace{16 \times 65536 \times 4096}_{\text{global layers}} + \underbrace{48 \times 4096 \times 4096}_{\text{local layers, capped}} = 4.75~\text{GiB}$$

A 3.4× reduction, and — more importantly — 75% of the layers now have a cache that does not grow with context at all.

The animation below shows both halves of the picture: the mask patterns themselves, and the receptive-field cone widening layer by layer under each interleaving scheme.

Figure 7 · Attention masks

Holes in the mask, cones through the depth

Frame 01 / 12

Implementation note

Mixing window sizes across layers means you need two attention masks in the same forward pass, and — if you follow Gemma 3 — two RoPE frequency tables. In PyTorch this typically means two BlockMask objects and a per-layer dispatch. It is a small amount of code that silently costs you a lot of throughput if you build the mask once per step instead of once per run.

One FFN became hundreds

Every change so far has been a refinement. Mixture-of-experts is a change of kind, and it is the single largest architectural difference between a 2020 model and a 2026 one.

The premise: for a dense model, parameters and FLOPs are welded together. Every parameter participates in every token. Scaling laws say more parameters is better; serving economics say more FLOPs per token is worse. MoE cuts the weld.

Replace the single FFN with \(N\) independent FFNs — "experts" — plus a small router. For each token, score the experts, take the top \(k\), and run only those:

$$g = \mathrm{softmax}\big(W_r x\big), \qquad \mathcal{T} = \operatorname{top-}k(g), \qquad y = \sum_{i \in \mathcal{T}} \tilde{g}_i \cdot \mathrm{FFN}_i(x)$$

Total FFN parameters scale with \(N\). FLOPs per token scale with \(k\). \(W_r\) is a single \(d \times N\) matrix — the router is essentially free.

The numbers this produces are startling, and the trend is toward more extreme sparsity:

ModelTotalActiveFractionExpertsTop-kShared
Mixtral 8×7B (2023)46.7B12.9B27.6%820
Qwen3-235B-A22B (2025)235B22B9.4%12880
gpt-oss-120b (2025)117B5.1B4.4%12840
DeepSeek-V3 (2024)671B37B5.5%25681
Kimi K2.5 (2026)1T32B3.2%38481
Qwen3-Coder-Next (2026)80B3B3.8%512101

Why many small experts beat few large ones

DeepSeekMoE's argument is combinatorial, and it is the cleanest first-principles case in this whole article. Fix the total expert parameter budget and the active budget. You can spend it as 8 experts with top-2, or 256 experts with top-8 where each expert is 32× smaller. Same parameters, same FLOPs. The difference is how many distinct routing configurations exist:

$$\binom{8}{2} = 28 \qquad\text{versus}\qquad \binom{256}{8} \approx 4.1 \times 10^{14}$$

Thirteen orders of magnitude more ways to combine specialists, at identical cost.

Finer granularity also reduces the pressure on any individual expert to be a generalist. With 8 experts each one must cover an eighth of everything the model does; with 256 an expert can plausibly specialize.

DeepSeek's second contribution is the shared expert: one expert that every token always uses, in addition to its top-\(k\). The reasoning is that some computation is genuinely universal — basic syntax, common formatting — and forcing every routed expert to independently relearn it wastes capacity. Adoption is split. DeepSeek, Kimi, GLM, and Qwen3-Next use a shared expert; Qwen3's main line and gpt-oss do not.

The problem MoE creates

Routing is learned, and learned routing collapses. If expert 7 is slightly better early in training, it gets more tokens, trains faster, gets more tokens. Left alone, a handful of experts absorb everything and the rest are dead weight.

The classical fix is an auxiliary load-balancing loss added to the objective, which is effective and also a gradient pointed away from the thing you actually want to minimize. DeepSeek-V3 introduced an auxiliary-loss-free alternative: keep a per-expert bias term that is added to routing scores only for the top-\(k\) selection, and adjust it up or down between steps based on observed load. Balance is enforced without ever touching the loss.

The systems consequence is larger than the modeling one. Experts do not fit on one device, so they are sharded — expert parallelism — and every MoE layer becomes an all-to-all: send each token to whichever devices hold its chosen experts, then gather the results back. Two all-to-alls per MoE layer, across 58 layers, over whatever fabric you have. On a cluster where the interconnect is much slower than NVLink, the routing pattern is the throughput.

Figure 8 · Mixture of experts

Parameters without FLOPs

Frame 01 / 12
Systems companionExpert parallelism, all-to-all, and how it composes with data, tensor, pipeline, and context parallelism — The Parallelism Playbook.

Past softmax: the 2026 hybrids

Everything in Section 7 accepts softmax attention and rations access to it. The 2026 releases start replacing it outright in most layers.

The opening is a reassociation. Write attention without the softmax normalizer, as a kernel:

$$o_m = \frac{\sum_{n \le m} \exp(q_m^\top k_n)\, v_n}{\sum_{n \le m} \exp(q_m^\top k_n)} \;\longrightarrow\; \frac{\sum_{n \le m} \phi(q_m)^\top \phi(k_n)\, v_n}{\sum_{n \le m} \phi(q_m)^\top \phi(k_n)}$$

Replace \(\exp(q^\top k)\) with a factorizable feature map \(\phi(q)^\top\phi(k)\).

Once it factorizes, matrix multiplication is associative and you can regroup:

$$\sum_{n \le m} \phi(q_m)^\top \phi(k_n) v_n \;=\; \phi(q_m)^\top \underbrace{\Big(\sum_{n\le m} \phi(k_n) v_n^\top\Big)}_{S_m,\;\text{a fixed } d\times d \text{ state}}$$

\(S_m = S_{m-1} + \phi(k_m)v_m^\top\) is a running sum. No cache. \(O(1)\) memory and \(O(1)\) work per token, forever.

The catch is exactly the thing that makes softmax attention good: \(\exp\) is sharp. It can concentrate almost all mass on one key, which is what "retrieve the value associated with this specific earlier token" requires. A fixed-size state that only ever accumulates cannot do precise retrieval — it superposes everything and interference grows.

Gated DeltaNet is the current best answer to that. Instead of a pure sum, the state is updated with a delta rule: write the new association, and subtract the old one first, with a learned gate controlling how much is erased and a decay controlling how fast old memories fade.

$$S_m \;=\; S_{m-1}\big(\alpha_m I - \beta_m k_m k_m^\top\big) \;+\; \beta_m v_m k_m^\top$$

\(\alpha\) decays, \(\beta\) controls the strength of the write. The \(kk^\top\) term removes the state's existing prediction for that key before writing the new one.

Nobody ships this alone. Qwen3-Next, Qwen3-Coder-Next, and Qwen3.5 use a 3:1 ratio of Gated DeltaNet blocks to gated softmax-attention blocks. Ant Group's Ling 2.5 does the same with Lightning Attention instead of DeltaNet. The reasoning mirrors sliding windows: most layers do not need precise long-range retrieval, and the handful that do can be full attention.

The third family is sparse selection. DeepSeek's Sparse Attention runs a cheap "indexer" that scores all previous tokens and keeps only the top few thousand per query, then does exact softmax attention over that subset. GLM-5 adopted it in 2026. The cache is unchanged; the compute is not.

Figure 9 · Beyond softmax

A growing cache versus a fixed state

Frame 01 / 10

Open question

There is no public apples-to-apples ablation — same data, same budget, same post-training — comparing a hybrid-attention model against a full-attention one at frontier scale. What we have is that Qwen moved hybrid attention from an experimental branch into their main line, which is a strong revealed preference but not a controlled experiment. Treat "linear hybrids match full attention" as a working hypothesis that several serious labs are betting on, not a settled result.

Small changes that aren't small

The vocabulary quadrupled, and it costs you memory you didn't budget

GPT-2 had 50,257 tokens. Llama 3 has 128,256. Qwen3 has 151,936. Gemma 3 has 262,144. The driver is coverage: a tokenizer trained mostly on English fragments Hindi, Arabic, and CJK into many tokens per word, which wastes context and compute on exactly the users you were trying to serve. Sarvam's team rebuilt the tokenizer from scratch and reported a 4× token-efficiency gain on Indian languages.

The cost is not in the embedding table — that is \(Vd\), a rounding error at 671B. The cost is the logits. Cross-entropy needs a tensor of shape \((B, S, V)\), and it is conventionally computed in fp32:

$$8 \times 8192 \times 151{,}936 \times 4~\text{bytes} \;\approx\; 37~\text{GiB}$$

One tensor. Before the backward pass allocates its twin.

This is why fused and chunked cross-entropy kernels went from a nice-to-have to a requirement. You compute logits for a slice of the sequence, take the loss, discard, repeat — never materializing the full tensor. If you are training a large-vocabulary model and your memory profile has an unexplained cliff at the loss, this is it.

Weight tying came back, for small models only

Tying the input embedding to the output projection saves \(Vd\) parameters. At 671B that is irrelevant; at 0.6B, where the embedding can be a third of the model, it is not. So Gemma 3 and Qwen3-0.6B tie, and everything above a few billion parameters does not. The reported trade-off is consistent: tying reduces parameters and slightly worsens loss.

Biases went away, then came partly back

Modern models drop the bias vector from every linear layer, including QKV projections. The argument is that after a normalization layer with its own learned scale, an additive bias is largely redundant, and the parameters are better spent elsewhere. Then gpt-oss shipped with attention biases, which surprised people. A 2023 analysis shows the key bias is provably redundant, and the empirical difference with and without is small — so this looks like a preserved internal convention rather than a finding.

Dropout is gone

Dropout was designed for many-epoch training on small datasets. Frontier pretraining is a single pass over trillions of tokens; the model never sees an example twice, so there is nothing to overfit. Removing it is free speed. A 2025 study on Pythia-scale models found dropout actively hurts in single-epoch regimes.

Attention sinks

Softmax forces the attention weights to sum to one. If nothing in the context is relevant to a query, the head still has to put its mass somewhere — and empirically it dumps it on the first token, which becomes a de facto null destination. This is fine until you evict early tokens for long-context streaming, at which point the head's dumping ground disappears and behaviour degrades.

gpt-oss handles this with a learned per-head logit appended to the softmax denominator:

$$a_n = \frac{\exp(z_n)}{\exp(\sigma_h) + \sum_{j} \exp(z_j)}$$

\(\sigma_h\) is a learned scalar per head. Mass can leave the sequence entirely; the weights over real tokens no longer have to sum to one.

Trinity Large reaches the same goal differently, with a sigmoid output gate on the attention result before the residual add — if the head has nothing to say, the gate closes.

Multi-token prediction

Attach small extra heads that predict tokens \(t{+}2, t{+}3, \dots\) alongside the usual \(t{+}1\), and sum the losses. It is a denser training signal for the same forward pass. DeepSeek-V3 used one extra head during training and made it optional at inference; Step 3.5 Flash and GLM-4.7 use three and keep them at inference, where the extra heads double as a built-in speculative decoder.

Width versus depth

gpt-oss-20b and Qwen3-30B-A3B are close in size and opposite in shape: 24 layers of width 2880 versus 48 layers of width 2048. Deeper models are more expressive per parameter but harder to train and impossible to parallelize along depth; wider models are faster to serve. GLM-5 went the other way from its predecessor, cutting layers from 92 to 78 while widening, presumably for latency. The only controlled comparison I know of is Gemma 2's Table 9, which found a wider 9B slightly ahead of a deeper one — a single data point, at one scale.

Build a model

All of the above is easier to feel than to read. The panel below implements the parameter, cache, and FLOP arithmetic from this article. Pick a preset to load a real configuration, then move the dials and watch which numbers move together.

The formulas it uses, in full, so you can check it:

$$P_\text{attn} = 2\,L\,d\,d_h(n_q + n_\text{kv}), \qquad P_\text{ffn}^\text{dense} = 3\,L\,d\,d_\text{ff}, \qquad P_\text{ffn}^\text{moe} = 3\,L\,d\,d_e(N + N_\text{shared})$$ $$P_\text{active} = P_\text{attn} + 3\,L\,d\,d_e(k + N_\text{shared}), \qquad \text{FLOPs/token} \approx 2 P_\text{active} + 4 L\, n_q d_h s$$

Embeddings counted once if tied, twice if not. The second FLOP term is attention scores and value aggregation, which the first term misses because it has no weights.

Figure 10 · Interactive

Configuration → parameters, cache, FLOPs

MLA presets use the latent-cache formula; GQA presets use the standard one. Attention FLOPs assume full attention at the stated sequence length and ignore the sliding-window discount.

The spec sheet

Nine years of drift, in one table. Read down a column to see how quickly a convention hardened.

ModelNormPositionFFNAttentionVocabContext
The original
Transformer (2017)LayerNorm, postsinusoidal, additiveReLU, 2 mats, 4dMHA, 8 heads37k BPE512
GPT-2 XL (2019)LayerNorm, prelearned absoluteGELU, 2 mats, 4dMHA, 25 heads50,2571,024
GPT-3 175B (2020)LayerNorm, prelearned absoluteGELU, 2 mats, 4dMHA, alternating banded50,2572,048
The consolidation
Llama 2 7B (2023)RMSNorm, preRoPE, b=104SwiGLU, 11008 = 8d/3MHA, 32 heads32,0004,096
Mistral 7B (2023)RMSNorm, preRoPESwiGLUGQA 32:8, SWA 409632,0008,192
Llama 3 8B (2024)RMSNorm, preRoPE, b=5×105SwiGLU, 14336 = 3.5dGQA 32:8128,2568k → 128k
The sparse turn
DeepSeek-V3 (2024)RMSNorm, preRoPE, decoupledMoE 256+1, top-8MLA, r=512, 128 heads129,280128k
Qwen3-235B-A22B (2025)RMSNorm + QK-normRoPE, b=106MoE 128, top-8GQA 64:4151,93632k → 131k
gpt-oss-120b (2025)RMSNorm + QK-normRoPE + sinksMoE 128, top-4GQA 64:8, SWA 128 at 1:1201,088131k
Gemma 3 27B (2025)RMSNorm, sandwichdual RoPE baseGeGLUGQA, SWA 1024 at 5:1262,144128k
Olmo 3 32B (2025)RMSNorm, post-branchRoPE, b=5×105, YaRNSwiGLUGQA 40:8, SWA 4096 at 3:1~100k65,536
The hybrid turn
Trinity Large (2026)RMSNorm ×4, depth-scaledRoPE local, NoPE globalMoE, DeepSeek-stylegated GQA, SWA 4096 at 3:1—256k
Kimi K2.5 (2026)RMSNormRoPE, decoupledMoE 384+1, top-8MLA—256k
Qwen3.5-397B-A17B (2026)RMSNorm + zero-centred QKpartial RoPEMoE + sharedGated DeltaNet : gated attn, 3:1—262k
GLM-5 (2026)RMSNormRoPEMoE 256+1, top-8MLA + sparse attention——

Dashes mark values I could not verify from a primary source at time of writing. Configuration details for the 2026 releases come from technical reports and model-hub configs; treat the last block as the most likely to have drifted.

What didn't change

An inventory of replacements can leave the impression that everything is in motion. It isn't, and the fixed points are informative.

Softmax attention still anchors every model. Even the hybrids keep one full attention layer in four, and the reason is specific: exact retrieval of a particular earlier token. Every attempt to remove it entirely has, so far, given that capability up.

The block is still attention-then-MLP with residual adds. Nine years, and the alternation between "mix across positions" and "compute within a position" has not been improved on. Even the parallel-block variants in Command-A and Tiny Aya keep both operations; they just stop making one wait for the other.

It is still next-token prediction with cross-entropy. Multi-token prediction adds heads; it does not change the objective. Every reasoning model in the 2026 table is a next-token predictor with post-training on top.

Decoder-only won completely. The 2017 paper was an encoder-decoder. The encoder-decoder split survives in translation and some multimodal connectors, but no frontier language model uses it.

The FLOP rule still holds. Forward-plus-backward is still roughly \(6N\) FLOPs per token — you just have to read \(N\) as active parameters now. That substitution is the single most important thing MoE changed about how you budget a training run.

And the largest fixed point is the one from the introduction. Every architecture in the final table is derived from the same GPT design, they differ by a handful of efficiency choices, and their benchmark differences track parameter count and training recipe far more closely than they track any of those choices. The architecture work of the last three years bought throughput, context length, and serving cost. It is not what made the models smart.

Which is exactly the question the next article in this series takes up: given that these models are enormously overparameterized relative to anything classical learning theory would sanction, why do they generalize at all?

References

Grouped by topic. Every entry links to its primary source.

The baseline architecture

Normalization

Position

Feed-forward and activations

Attention efficiency

Mixture of experts

Beyond softmax attention

Model reports referenced in the spec sheet

Surveys and comparisons