Why Neural Networks Need Nonlinearity

Every neural network you have ever trained is a stack of linear maps with something small and strange wedged between them. That something is a handful of scalar operations per element — the cheapest part of the whole forward pass, a rounding error in the FLOP budget — and yet swapping it out has repeatedly changed what was trainable at all. ReLU made depth practical in 2011. GELU quietly became the transformer default. SwiGLU took over frontier language models around 2022 and never left. This is a field guide to that lineage: what each function does, why it was proposed, what breaks, and who ships what.

I'll assume you know backprop. Everything else — the saturation argument, the sparsity argument, the gating argument — gets built from scratch, with figures you can step through frame by frame. Nine of them are animated; all of them have Previous / Next controls if you would rather read them like a comic strip than watch them.

01
Why anything at all

Compose two affine maps and you get an affine map. W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂). Stack a hundred of them and you still have a single matrix, wasting ninety-nine layers of parameters on a function a one-layer model could express exactly. The nonlinearity is what makes depth mean anything.

That much is textbook, and it is also where most explanations stop — which is misleading, because it suggests the choice of nonlinearity barely matters. The universal approximation theorems are permissive: Cybenko (1989) and Hornik (1991) require only that the function be non-polynomial (plus mild regularity), and Leshno et al. (1993) sharpened this to essentially any non-polynomial continuous function. By that standard sigmoid, ReLU and x³ − x are equally good.

They are not equally good. Approximation theory tells you what a network can represent given ideal weights; it says nothing about whether gradient descent will find those weights, how the signal scales as it propagates, how much activation memory you burn, or how the intermediate tensors quantize. Almost the entire history of activation function research is about those four things, not about expressivity. Keep that framing and the timeline stops looking like fashion.

Here is the cast. Step through it — the camera zooms into the origin on the piecewise-linear members, because that is where their whole personality lives.

FIG 01

The atlas: twelve functions, one continuous morph

Solid line is f(x), dashed line is f′(x). Watch the derivative, not the function — the derivative is what the optimizer sees. Note how the saturating pair (sigmoid, tanh) have derivatives that die at both ends, while the rectifier family keeps a derivative of exactly 1 on the whole positive half-line.

Four properties do most of the explanatory work in what follows, so it's worth naming them:

Saturation. Does f′ go to zero for large |x|? Two-sided saturation is what killed the sigmoid era.
Zero-centring. Is the output mean near zero? If not, the gradient with respect to every weight in a row shares a sign, and updates zig-zag (LeCun et al., Efficient BackProp, 1998).
Smoothness. Is f differentiable at 0, and is f′ continuous? This matters far less than people assume for SGD, but a lot for second-order methods, for loss-landscape sharpness, and for whether small logit perturbations move smoothly.
Self-gating. Does the function multiply its input by a data-dependent scalar in [0, 1]? This is the property that quietly links GELU, Swish, and the entire GLU family, and it is the through-line of the modern era.

02
The saturating era, and the wall it hit

The 1986 backprop paper used the logistic sigmoid, σ(x) = 1/(1 + e⁻ˣ), and it was a reasonable choice: bounded, smooth, monotone, with the elegant derivative σ′ = σ(1 − σ) and a tidy probabilistic reading as a firing rate. LeCun's Efficient BackProp then made the case for tanh instead — same shape, but zero-centred, with a derivative of 1 at the origin rather than 0.25, which keeps the forward variance closer to unity.

Both saturate on both sides, and that is fatal at depth. The maximum of σ′ is 0.25. Even in the best case, every layer you pass through on the way back multiplies the gradient by at most a quarter, and typically much less once you account for the distribution of pre-activations. Hochreiter's 1991 thesis and Bengio et al. (1994) formalised this; the practical consequence is that the early layers of a deep sigmoid network receive a gradient many orders of magnitude smaller than the last layers, and effectively never train.

FIG 02

Watching a gradient die

A gradient pulse travels backward through 18 layers. Bars are log₁₀ of the gradient norm relative to the loss — each row drops by roughly a constant factor per layer, so a straight diagonal on this plot is exponential decay. Sigmoid with Xavier initialisation loses about 0.65 decades per layer; ReLU with He initialisation holds its own.

Notice what the fix is not. Nothing about sigmoid limits what the network can represent. The problem is entirely about the conditioning of the optimisation, which is exactly the framing from section 01.

Two fixes arrived at once, and it is worth being honest that the activation function was only one of them. Better initialisation (Glorot & Bengio 2010; He et al. 2015), then normalisation layers (Ioffe & Szegedy 2015; Ba et al. 2016) and residual connections (He et al. 2015) attacked the same problem from other directions. Today, with pre-norm residual transformers, you could probably train a tanh network to convergence. The activation function stopped being the binding constraint years ago — which is precisely why the modern choices are made on subtler grounds like sparsity, memory and quantization behaviour.

03
The rectifier revolution

ReLU(x) = max(0, x). Fukushima's cognitron used something like it in 1969, Nair & Hinton (2010) motivated it as an infinite stack of shifted binary units in an RBM, Glorot, Bordes & Bengio (2011) made the deep-learning case in Deep Sparse Rectifier Neural Networks, and AlexNet (2012) made it non-negotiable by reporting a six-fold speedup to a given training error on CIFAR-10 versus tanh.

What ReLU actually buys you:

An exactly-1 derivative on the positive half-line. No attenuation, no amplification. The gradient path through active units is a pure identity, which is why very deep rectifier nets train.
Genuine sparsity. Around half the units are exactly zero at initialisation, and the exact zero matters — it is not a small number, it is a structural absence. Glorot et al. argued this gives disentangled, information-efficient representations.
Near-free compute. A comparison and a select. No exponentials, no divisions.
Piecewise linearity. The whole network becomes a continuous piecewise-linear function, which is what makes exact analyses (ReLU network polytope counting, LRP-style attribution, some verification methods) tractable.

The cost is the dying ReLU. If a unit's pre-activation is negative for every input in the data distribution, its gradient is exactly zero for every input, forever. There is no slow recovery — it is an absorbing state. A large learning rate that pushes a bias sharply negative, or a poorly scaled input, can kill a substantial fraction of a layer in a few hundred steps.

FIG 03

Dying units are an absorbing state

60 units per panel, tracked over 1,200 simulated steps under identical negative bias drift. Cell brightness is firing rate; black cells are permanently dead. The leaky panel never loses a unit, because a gradient of 0.1 on the negative side is still a gradient. The inset follows one unit's pre-activation distribution as it slides below zero.

The patch kit

Everything in this family keeps the positive half-line identity and argues about the negative half.

Leaky ReLU f(x) = x if x > 0 else αx, α = 0.01 Maas, Hannun & Ng 2013 PReLU same, but α is a learned per-channel parameter He et al. 2015 RReLU α sampled uniformly per unit during training Xu et al. 2015 ELU f(x) = x if x > 0 else α(eˣ − 1) Clevert et al. 2015 SELU λ·ELU(x) with λ=1.0507, α=1.6733 Klambauer et al. 2017 ReLU6 min(max(0, x), 6) Krizhevsky 2010 / MobileNet

PReLU is the historically important one: the same paper introduced He initialisation and was the first to report super-human top-5 accuracy on ImageNet. Its learned α values are instructive — early conv layers learn α near 0.25 (keeping negative information, behaving almost linearly), while deeper layers drive α toward 0 (becoming more discriminative, more ReLU-like).

ELU pushes the negative saturation value to −α, which pulls the mean activation toward zero and gives a "natural gradient"-like effect without batch statistics. SELU goes further: with those two very specific constants, LeCun-normal initialisation and alpha-dropout, the map from layer input to layer output has a stable fixed point at zero mean and unit variance — self-normalising, with a 90-page appendix proving it. It is beautiful and almost nobody uses it, because layer norm is more robust and does not constrain the architecture.

ReLU6 exists for a purely engineering reason: capping activations at 6 keeps them in a range that survives 8-bit fixed-point quantization, which is why it appears throughout the MobileNet line.

04
Smooth gates: GELU, Swish, Mish

The next move was to make the gate soft. Instead of ReLU's hard indicator x · 1[x > 0], multiply x by something that transitions smoothly from 0 to 1.

Hendrycks & Gimpel's GELU (2016) derived this from a stochastic-regularisation argument rather than by curve-fitting, and the derivation is genuinely worth understanding. Consider a gate that zeroes each unit with a probability that depends on how large the unit is: keep x with probability Φ(x), the standard normal CDF. That is dropout where the drop rate is a function of the input rather than a fixed hyperparameter — inputs that are large relative to the layer's activation distribution are kept, small or negative ones are usually zeroed. Take the expectation of this stochastic gate and you get a deterministic function:

GELU(x) = x · Φ(x) = x · ½[1 + erf(x/√2)] tanh approximation: 0.5x(1 + tanh[√(2/π)(x + 0.044715x³)]) sigmoid approximation: x · σ(1.702x) ← "QuickGELU", used by OpenAI CLIP

FIG 04

GELU as the expectation of a coin flip

Left: keep-probability Φ(x), with live Bernoulli draws at 41 sample points. Right: the Monte-Carlo estimate of E[x · Bernoulli(Φ(x))] as the number of draws grows from 1 to 512. The scatter collapses onto the exact GELU curve; ReLU's hard gate is dashed for reference. GELU is not a smoothed ReLU that someone drew — it is the mean of an adaptive dropout mask.

Swish arrived from the other direction. Ramachandran, Zoph & Le (2017) ran a reinforcement-learning architecture search over compositions of unary and binary primitives and the search returned x · σ(βx). With β = 1 this is exactly the SiLU that Elfwing, Uchibe & Doya had proposed in 2017 as the derivative of a softplus-like unit in a reinforcement learning context, and it is also almost exactly GELU — the two curves differ by under 0.02 everywhere. β is sometimes learned; at β → ∞ Swish becomes ReLU, at β → 0 it becomes the identity scaled by a half, so β interpolates between linear and rectified.

Mish (Misra, 2019) is x · tanh(softplus(x)), slightly smoother and with a slightly deeper negative dip. It shows up in the YOLO detection line and not much else, mostly because it costs more and the gains are within noise on most benchmarks.

Hard-swish, from MobileNetV3, replaces the sigmoid with a piecewise-linear ramp: x · ReLU6(x + 3)/6. Same shape, no exponential, quantizes cleanly. This is the recurring mobile-vision pattern — take the good curve, approximate it with clamps.

All of these share a property that ReLU lacks and that turns out to matter: they are non-monotone. There is a small negative dip near x ≈ −1.28 for GELU (minimum value ≈ −0.17). A unit can be slightly negative and still contribute a small negative signal, and crucially the derivative there is negative rather than zero, so the unit can be pushed back toward the active region. There is no absorbing state.

FIG 05

The derivative is what the optimizer feels

A tangent sweeps from x = −5 to x = +5 while the lower panel tracks f′(x). Look for three things: how wide the region with usable gradient is, whether f′ exceeds 1 (GELU and SiLU both overshoot to ≈1.1, mildly amplifying gradients near x ≈ 1.5), and whether f′ ever goes negative.

05
Gated linear units, and why SwiGLU won

Everything so far is a scalar function applied elementwise. The GLU family breaks that rule: it splits the tensor in two, runs a nonlinearity on one half, and multiplies. The gate is computed from a different learned projection of the input than the value it modulates.

Dauphin et al. (2016) introduced this for convolutional language models as GLU(x) = (xW + b) ⊙ σ(xV + c), arguing that the linear path gives gradients an unattenuated route while the sigmoid path controls what passes. Shazeer's 2020 note GLU Variants Improve Transformer then swept the choice of gate nonlinearity inside the transformer feed-forward block:

Bilinear (xW) ⊙ (xV) no nonlinearity at all ReGLU ReLU(xW) ⊙ (xV) GEGLU GELU(xW) ⊙ (xV) SwiGLU SiLU(xW) ⊙ (xV) also written Swish₁(xW) ⊙ (xV) FFN(x) = [ SiLU(xW_gate) ⊙ xW_up ] W_down

Every gated variant beat the ungated baselines on T5's span-corruption pre-training loss and on downstream GLUE and SuperGLUE, by small but consistent margins. Shazeer offered no mechanistic explanation and attributed the result, memorably, to divine benevolence. Six years later there is still no fully satisfying theory, but there are two decent partial ones.

The first is multiplicative interaction. A gated unit computes a degree-2 polynomial in the input, so a single FFN layer can express products of features — feature conjunctions — that an elementwise nonlinearity would need depth to build. Even the bilinear variant, with no nonlinearity whatsoever, outperforms a ReLU FFN in Shazeer's sweep, which strongly suggests the multiplication is doing the work rather than the shape of the gate curve.

The second is selectivity. The gate is a learned, input-dependent mask over the hidden dimension, which is a soft form of conditional computation — a continuous relative of mixture-of-experts routing living inside every FFN.

FIG 06

Building a SwiGLU block, one edge at a time

From a two-matmul vanilla FFN to a three-matmul gated FFN, with the parameter budget rebalanced at the end. Real dimensions from Llama-2-7B (d = 4096, h = 11008) are used throughout so the arithmetic is checkable.

The 8d/3 rule, and why 11008

A vanilla FFN with hidden width h has 2dh parameters. A gated FFN has 3dh, because of the extra gate projection. To keep the parameter and FLOP budget fixed when switching to a gated block, you shrink h by a factor of 2/3. With the conventional h = 4d, that gives:

h = (2/3) · 4d = 8d/3 ≈ 2.667d Llama-2-7B: d = 4096 → 8d/3 = 10922.7 → rounded to a multiple of 256 → 11008 Llama-3-8B: d = 4096 → 14336 (3.5d — they deliberately spent extra on the FFN) Mistral-7B: d = 4096 → 14336 Qwen2.5-7B: d = 3584 → 18944 (5.3d — very wide, paired with GQA savings) Gemma-2-9B: d = 3584 → 14336 (GeGLU, not SwiGLU)

Those odd numbers in config files are not arbitrary — they are 8d/3 rounded up to a hardware-friendly multiple, or a deliberate departure from it. Note too that the trend since Llama-3 has been to spend more than 8d/3 on the FFN, because GQA freed up parameter budget from the attention block and wide FFNs are the most GPU-efficient place to put capacity.

FIG 07

What gating looks like in two dimensions

The output surface over the (value, gate) plane. An elementwise activation ignores one axis entirely — its contours are vertical stripes. A gated unit's contours bend, because the gate rescales the value path rather than replacing it. The final stage sweeps a horizontal slice to show the induced 1-D function changing shape as the gate moves.

The engineering costs nobody mentions

Three matmuls, not two. The gate and up projections can be fused into one [d, 2h] GEMM, which most efficient implementations do — but that fusion changes the weight layout, which is why converting checkpoints between frameworks so often gets the gate and up halves swapped. Symptom: the model runs, loss is finite, outputs are garbage.
More activation memory. You must keep both SiLU(xW_gate) and xW_up for the backward pass, or recompute. Two tensors of shape [B, T, h] instead of one. Under selective activation checkpointing this is usually the single biggest FFN line item, and recomputing the whole gated block is cheap relative to attention, which is why "recompute MLP, keep attention" is a common recipe.
Outliers. The elementwise product of two learned projections has a much heavier tail than a post-ReLU tensor. Emergent activation outliers concentrate in the input to W_down, which is exactly the tensor that per-tensor INT8 quantization handles worst. SmoothQuant and friends exist largely because of this.

06
ReLU strikes back: sparsity as a system property

The smooth-gate era had a hidden cost: GELU and SiLU produce almost no exact zeros. Under ReLU, roughly half of the FFN intermediate tensor is structurally zero, and the corresponding columns of W_down never need to be read from memory. In a memory-bandwidth-bound decoder — which is exactly what single-batch inference is — that is a free 2× on the FFN.

Mirzadeh et al. (2023), ReLU Strikes Back, made this case directly: substituting ReLU for GELU in pre-trained LLMs costs little in quality after a short fine-tune and yields large inference savings. ProSparse and the TurboSparse line push activation sparsity to 90%+ with explicit sparsity-inducing regularisation. And Nemotron-4 took the other route — never leave ReLU at all — using squared ReLU throughout:

ReLU²(x) = max(0, x)² Primer (So et al. 2021), Nemotron-4, Persimmon / Fuyu, RWKV channel-mix Derivative: 2·max(0, x) — unbounded, so it needs careful init and often qk-norm elsewhere

Squared ReLU keeps hard sparsity while restoring some of the multiplicative flavour of a gate: it is literally x ⊙ ReLU(x) with the value and gate paths tied to the same projection. That framing makes it obvious what SwiGLU adds — untied projections — and why squared ReLU is nearly free in comparison.

FIG 08

Where the zeros go

Post-activation histograms over 6,000 pre-activations drawn from a heavy-tailed layer distribution. Exact-zero fraction is the bar that matters for kernel-level skipping; the last stage sweeps a threshold ε to show that GELU and SiLU are nearly sparse — plenty of near-zero mass, but no structural zeros a kernel can exploit without a predictor.

07
Who ships what

This is the part that is genuinely hard to look up, because it is buried in config files and rarely stated in papers. Filter by family:

Four patterns fall out of that grid.

The 2020 dividing line. Anything designed before roughly 2021 uses GELU; anything designed after uses a gated variant. BERT, GPT-2, GPT-3, ViT, BLOOM, OPT, Falcon, GPT-NeoX — all elementwise. PaLM, Llama, Mistral, Qwen, Gemma, DeepSeek, OLMo — all gated. PaLM is the paper that made SwiGLU the default.

SwiGLU versus GeGLU is a coin flip. Google's line (T5 v1.1, Flan-T5, Gemma 1/2/3) uses GeGLU; nearly everyone else uses SwiGLU. Given that SiLU and GELU differ by less than 0.02 pointwise, this is almost certainly a lineage artefact rather than a measured decision.

Vision-language models are chimeras. A typical VLM runs QuickGELU inside the vision tower, plain GELU in the connector MLP, and SwiGLU in the language model — three different activation functions in one forward pass. LLaVA-1.5 and the original Molmo release are exactly this. Qwen2.5-VL is notable for breaking the pattern by rebuilding its ViT with SwiGLU and RMSNorm to match its language model.

The squared-ReLU pocket is real but small. Nemotron, Persimmon/Fuyu, Primer and RWKV's channel-mix all use it, mostly for inference-time sparsity and mostly at organisations that control their own serving stack.

08
Nuances that will cost you a day

QuickGELU is not GELU, and CLIP cares

OpenAI's CLIP implementation defines its activation as x * sigmoid(1.702 * x). The weights were trained with that function. OpenCLIP's default ViT-L-14 uses true GELU, and you must ask for the -quickgelu variant to match the original checkpoints. The two curves differ by at most about 0.02 — which sounds harmless and is not. Load OpenAI CLIP weights into a plain-GELU tower and zero-shot ImageNet accuracy drops by roughly a point, image-text retrieval degrades, and nothing crashes. If you are building a VLM on CLIP ViT-L/14-336 — the backbone in LLaVA, Molmo and many others — check this before you check anything else. SigLIP and SigLIP 2 sidestep the whole issue by using the tanh-approximate GELU consistently.

Debugging heuristic

A VLM whose vision tower is "slightly worse than reported" but not broken is almost always one of three things: a GELU/QuickGELU mismatch, an image-normalisation constant mismatch (CLIP's mean/std versus ImageNet's versus SigLIP's simple −1..1), or an interpolation-mode mismatch in the resize. All three cost about a point and none of them raise an exception.

The tanh approximation is a different function

PyTorch's nn.GELU() defaults to the exact erf form; nn.GELU(approximate='tanh') gives the polynomial. JAX and the original TF BERT code used the tanh form. The maximum discrepancy is around 0.0003, which is irrelevant for training from scratch and occasionally relevant when you are chasing a bit-exact reproduction across frameworks or debugging a numerical divergence between a reference implementation and a fused kernel. Gemma explicitly specifies the tanh approximation in its config, which is a good habit.

Clamped gates for stability

SwiGLU's output is a product of two unbounded projections, so it can produce very large values, and squared ReLU is worse. At large scale several groups have started clamping the gated activation to a fixed range as a cheap stability measure — gpt-oss ships a SwiGLU variant with clamping and a sigmoid coefficient of 1.702, which makes its gate a QuickGELU in all but name. If you see loss spikes localised to the FFN, clamping the pre-multiply tensors is a much cheaper experiment than changing the optimizer.

Initialisation must match the activation

tanh / sigmoid → Xavier/Glorot, gain 1 Var(W) = 2/(fan_in + fan_out) ReLU family → He/Kaiming, gain √2 Var(W) = 2/fan_in SELU → LeCun normal + AlphaDropout Var(W) = 1/fan_in (required, not advisory) GELU / SiLU → He is close enough effective gain ≈ 1.7 rather than √2 ≈ 1.41

The √2 in He initialisation exists precisely because ReLU discards half the variance. Use Xavier with ReLU and your forward activations shrink by √2 per layer; use He with tanh and they grow. In a pre-norm transformer the normalisation layers hide most of this, which is why people get away with mismatches, but it still shows up as a worse first thousand steps.

Non-differentiability at zero is a non-issue

ReLU is not differentiable at 0. Frameworks pick a subgradient (PyTorch returns 0) and nothing bad happens, because the probability of landing exactly on 0 in floating point is negligible and the loss surface is still Lipschitz. The smoothness argument for GELU is real, but it is about the shape of the loss landscape away from the kink, not about the measure-zero point itself.

Learned activations exist and mostly don't pay

PReLU learns one parameter per channel. Swish can learn β. APL and PAU learn piecewise-linear or rational functions. KANs (2024) go all the way and replace every weight with a learned spline. The recurring finding is that learned activations help small models on small data and wash out at scale, where the network can absorb any fixed nonlinearity's limitations into its weights. The exception worth watching is xIELU, an activation with two learned slope parameters that appeared in the Apertus models — a rare case of a learned activation shipping in a frontier-scale open model.

09
Thirty-five years, one axis

FIG 09

The lineage

The camera pans across eight eras. Colour encodes family: saturating, rectifier, smooth gate, gated linear unit, sparsity revival.

10
What to actually use

Transformer LLM, training from scratch. SwiGLU with h = 8d/3 rounded to a multiple of 256. This is the default for a reason and the burden of proof is on any alternative.
You control the inference stack and are memory-bandwidth bound. Consider squared ReLU, or ReLU with a sparsity predictor. You are trading a small quality delta for a large decode-time win.
Vision encoder, training from scratch. GELU (tanh approximation, specified explicitly). If you are at ViT-g scale or above, SwiGLU FFNs are worth it — DINOv2's giant model and EVA-02 both went that way.
Vision encoder, loading pre-trained weights. Use whatever the checkpoint was trained with. Check for QuickGELU. Check twice.
Connector / projector MLP in a VLM. GELU. It is two layers and nobody has shown it matters; spend your experiments elsewhere.
Small conv nets, mobile, quantized. Hard-swish or ReLU6. The clamps are the point.
An RNN or anything with recurrent state. tanh and sigmoid remain correct — bounded outputs are what keeps the state from exploding across timesteps. Mamba's SiLU gate is the modern exception, and it works because the state update is structurally stable by construction.
Fine-tuning anything. Do not change the activation function. Non-negotiable.

The meta-lesson: activation functions stopped being about expressivity a long time ago, and stopped being about trainability once normalisation and residuals arrived. What remains is a systems question — sparsity, memory, quantization tails, kernel fusion — and that is why the answer keeps moving. It will move again when the hardware does.

11
References

Grouped by what each paper is actually about rather than by date. Every entry links out to the paper, the proceedings page, or — where the choice only exists in code — the implementation. The tag on the right of each line is the activation the paper is a primary source for.

Foundations and theory

Why a nonlinearity is needed at all, and why that argument says nothing about which one to pick.

1980Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position Fukushima · Biological Cyberneticsrectifier, avant la lettre
1986Learning representations by back-propagating errors Rumelhart, Hinton & Williams · Naturesigmoid
1989Approximation by superpositions of a sigmoidal function Cybenko · Mathematics of Control, Signals and Systemsuniversal approximation
1991Approximation capabilities of multilayer feedforward networks Hornik · Neural Networksuniversal approximation
1991Untersuchungen zu dynamischen neuronalen Netzen Hochreiter · diploma thesis (the vanishing gradient)why saturation kills depth
1993Multilayer feedforward networks with a non-polynomial activation function can approximate any function Leshno, Lin, Pinkus & Schocken · Neural Networksnon-polynomial suffices
1994Learning long-term dependencies with gradient descent is difficult Bengio, Simard & Frasconi · IEEE Trans. Neural Networksgradient decay
1998Efficient BackProp LeCun, Bottou, Orr & Müller · Neural Networks: Tricks of the Tradetanh over sigmoid

The saturating era and the other fixes

Initialisation, normalisation and residuals attacked the same problem from different directions — which is why the activation stopped being the binding constraint.

2010Understanding the difficulty of training deep feedforward neural networks Glorot & Bengio · AISTATSXavier init
2015Batch normalization: accelerating deep network training by reducing internal covariate shift Ioffe & Szegedy · ICMLcontext
2015Deep residual learning for image recognition He, Zhang, Ren & Sun · CVPRcontext
2016Layer normalization Ba, Kiros & Hintoncontext

The rectifier family

Hard gates: the positive half-line identity, and thirty years of arguing about the negative half.

2010Rectified linear units improve restricted Boltzmann machines Nair & Hinton · ICMLReLU
2011Deep sparse rectifier neural networks Glorot, Bordes & Bengio · AISTATSReLU
2012ImageNet classification with deep convolutional neural networks Krizhevsky, Sutskever & Hinton · NeurIPSReLU, 6× speedup
2013Rectifier nonlinearities improve neural network acoustic models Maas, Hannun & Ng · ICML workshopLeaky ReLU
2015Delving deep into rectifiers: surpassing human-level performance on ImageNet classification He, Zhang, Ren & Sun · ICCVPReLU + He init
2015Fast and accurate deep network learning by exponential linear units (ELUs) Clevert, Unterthiner & Hochreiter · ICLRELU
2015Empirical evaluation of rectified activations in convolutional network Xu, Wang, Chen & LiRReLU
2017Self-normalizing neural networks Klambauer, Unterthiner, Mayr & Hochreiter · NeurIPSSELU

Smooth self-gating

Replace the hard indicator with a soft, input-dependent multiplier in [0, 1].

2001Incorporating second-order functional knowledge for better option pricing Dugas, Bengio, Bélisle, Nadeau & Garcia · NeurIPSsoftplus
2016Gaussian error linear units (GELUs) Hendrycks & GimpelGELU
2017Sigmoid-weighted linear units for neural network function approximation in reinforcement learning Elfwing, Uchibe & DoyaSiLU
2017Searching for activation functions Ramachandran, Zoph & LeSwish
2019Mish: a self regularized non-monotonic activation function Misra · BMVCMish
2019Searching for MobileNetV3 Howard et al. · ICCVhard-swish

Gated linear units

Two projections, one multiplied into the other. The current frontier default, still without a satisfying theory.

2016Language modeling with gated convolutional networks Dauphin, Fan, Auli & Grangier · ICMLGLU
2019Exploring the limits of transfer learning with a unified text-to-text transformer Raffel et al. · JMLR (T5; v1.1 switched to GEGLU)ReLU → GEGLU
2020GLU variants improve transformer ShazeerGEGLU, SwiGLU

Sparsity and the squared rectifier

The counter-move: give up smoothness, get exact zeros a kernel can skip.

2021Primer: searching for efficient transformers for language modeling So, Mańke, Liu, Dai, Shazeer & Lesquared ReLU
2023ReLU strikes back: exploiting activation sparsity in large language models Mirzadeh et al. · AppleReLU
2024ProSparse: introducing and enhancing intrinsic activation sparsity within large language models Song et al.ReLU

What ships: language models

Papers whose configs pin down a choice. These are the primary sources behind the adoption matrix above.

2018BERT: pre-training of deep bidirectional transformers for language understanding Devlin, Chang, Lee & Toutanova · NAACLGELU
2020Language models are few-shot learners Brown et al. · NeurIPS (GPT-3)GELU
2022OPT: open pre-trained transformer language models Zhang et al. · MetaReLU
2022PaLM: scaling language modeling with Pathways Chowdhery et al. · GoogleSwiGLU
2023LLaMA: open and efficient foundation language models Touvron et al. · MetaSwiGLU
2023Llama 2: open foundation and fine-tuned chat models Touvron et al. · MetaSwiGLU, h = 11008
2023Mistral 7B Jiang et al. · Mistral AISwiGLU
2023Mamba: linear-time sequence modeling with selective state spaces Gu & DaoSiLU gate
2024Gemma 2: improving open language models at a practical size Gemma Team · Google DeepMindGeGLU (tanh)
2024Nemotron-4 340B NVIDIAsquared ReLU
2024DeepSeek-V3 technical report DeepSeek-AISwiGLU
20252 OLMo 2 Furious OLMo Team · Ai2SwiGLU

What ships: vision and vision-language

Where the mismatches hide. If you are debugging a VLM tower, start here.

2020An image is worth 16×16 words: transformers for image recognition at scale Dosovitskiy et al. · ICLR (ViT)GELU
2021Learning transferable visual models from natural language supervision Radford et al. · ICML (CLIP)QuickGELU
—open_clip — reference implementation and the -quickgelu model suffix Ilharco et al. · GitHubthe footgun, in code
2023Sigmoid loss for language image pre-training Zhai, Mustafa, Kolesnikov & Beyer · ICCV (SigLIP)GELU (tanh)
2023DINOv2: learning robust visual features without supervision Oquab et al. · MetaGELU; SwiGLU at ViT-g
2023EVA-02: a visual representation for neon genesis Fang et al.SwiGLU in the ViT
2024Molmo and PixMo: open weights and open data for state-of-the-art vision-language models Deitke et al. · Ai2QuickGELU + SwiGLU
2025Qwen2.5-VL technical report Qwen Team · AlibabaSwiGLU in the ViT too
2025SigLIP 2: multilingual vision-language encoders with improved semantic understanding, localization, and dense features Tschannen et al. · Google DeepMindGELU (tanh)

Learned and searched activations

Let the network pick the shape. Helps small models; mostly washes out at scale.

2014Learning activation functions to improve deep neural networks Agostinelli, Hoffman, Sadowski & Baldi (APL)piecewise-linear, learned
2024KAN: Kolmogorov–Arnold networks Liu et al.learned splines on edges

Systems consequences

Why the gated product's fat tail is an inference problem, not just a curiosity.

2022LLM.int8(): 8-bit matrix multiplication for transformers at scale Dettmers, Lewis, Belkada & Zettlemoyer · NeurIPSactivation outliers
2022SmoothQuant: accurate and efficient post-training quantization for large language models Xiao, Lin, Seznec, Wu, Demouth & Han · ICMLFFN down-proj inputs
2024Massive activations in large language models Sun, Chen, Kolter & Liuwhere the outliers live

Model-to-activation mappings above were read from public configuration files and reference implementations where available. Closed models are marked undisclosed rather than guessed.