Across the CNNVerse: A Visual Guide to CNN Architectures

Almost everything interesting about convolutional networks follows from one small, stubborn idea: instead of learning a separate weight for every pixel position, learn a small pattern once and slide it everywhere. That single decision buys translation equivariance, cuts parameter counts by three or four orders of magnitude, turns a dense matrix multiply into a structured one, and — crucially — makes the whole thing stackable. Thirty years of architecture research is largely the story of what people did once they took that constraint seriously.

This guide is built in two halves. The first half is the operation itself: the arithmetic, the geometry, the receptive field, the reason the inductive bias works, and the family of variants that grew out of it — 1D, 2D, 3D, depthwise, grouped, dilated, transposed. The second half is the lineage: LeNet through ConvNeXt, each one analysed for what it changed, what that change cost in time and memory, and what it bought in accuracy. At the end, there's a rig where you push a single image through all nine architectures and watch their different spatial schedules chew through it.

Every figure here is computed, not illustrated. The parameter and MAC counts you'll see come from a layer-by-layer spec engine running live in the page, and they reproduce the published numbers to within a percent. The feature maps in the interactive sections are real convolutions running on real pixels in your browser.

Part oneThe operation

The operation

Start with the discrete two-dimensional case, one input channel, one filter. You have an input array X and a small weight array W of size k×k. The output at position (i, j) is a dot product between the filter and the patch of input that sits underneath it:

Y[i, j] = Σ_u=0..k-1 Σ_v=0..k-1 W[u, v] · X[i + u, j + v] + b ↑ this is cross-correlation, which is what every deep learning framework calls "convolution". True convolution flips the kernel: X[i - u, j - v]. Since W is learned, the flip is a relabelling and nobody bothers.

With C_in input channels the filter becomes a small volume of shape k×k×C_in and the sum runs over channels too. With C_out filters you stack C_out such volumes and get an output with C_out channels. That's the whole layer:

Y[i, j, o] = Σ_c Σ_u Σ_v W[u, v, c, o] · X[i·s + u - p, j·s + v - p, c] + b[o] parameters = k · k · C_in · C_out + C_out MACs = k · k · C_in · C_out · H_out · W_out output size = floor((H_in + 2p - k) / s) + 1

Notice which terms are in which formula. Parameter count does not depend on the spatial size of the input — that is the whole point of weight sharing. Compute cost does, and it multiplies. This asymmetry is the single most important fact for understanding why architectures look the way they do: early layers are cheap in parameters and ruinous in FLOPs, late layers are the reverse. Nearly every design decision in the second half of this article is a negotiation with that asymmetry.

Figure 01 · The sliding window

One multiply-accumulate at a time

A 3×3 filter walks a 6×6 input and fills a 4×4 output. The camera follows the window; each stage shows the nine products, their running sum, and where that sum lands.

Every value is computed live. Press Next to step one output cell at a time; the running sum on the right accumulates in the same order the arithmetic does.

Step through that figure once with the Next button and the shape rule stops being something you memorise. A 6×6 input with a 3×3 filter and no padding has exactly 4 valid positions per axis, because the window needs 3 cells and it can start at index 0, 1, 2 or 3. Every output pixel is one dot product of 9 terms. The layer as a whole is 16 dot products, sharing the same 9 weights.

What a filter sees

A filter is a template matcher. The dot product is maximised when the input patch looks like the filter, so a filter with a positive stripe above a negative stripe fires on horizontal edges with dark-on-top polarity, and goes negative on the opposite polarity. This is not a metaphor — it is literally cosine similarity up to a scale factor.

The classical image-processing kernels are the cleanest way to build the intuition, because you already know what they do. Then, once you've internalised that a 3×3 array of numbers is a visual concept, the leap to "and the network learns which numbers" is small.

Figure 02 · Kernel laboratory

Edit the nine numbers, watch the concept change

Pick a preset or type your own weights. The scan animation fills the output row by row so you can see the response build; hover the input to inspect the exact patch arithmetic at that location.

Convolution runs on the luminance channel at full resolution in your browser. Sobel-x, Sobel-y, Laplacian, box blur, Gaussian, sharpen and emboss are the presets; the identity kernel is there so you can start from scratch.

Three things are worth noticing while you play with it. First, sign structure carries orientation: swapping the positive and negative halves of Sobel-x flips which edges light up, and rotating the pattern by 90° turns a horizontal-edge detector into a vertical one. Second, the sum of the weights sets the DC response: kernels that sum to zero are blind to flat regions and respond only to change, which is why every edge detector sums to zero and every blur sums to one. Third, magnitude is gain: doubling every weight doubles the response, which is exactly why normalisation layers matter so much later on.

The learned version is not so different

If you visualise the first-layer filters of a trained AlexNet, roughly half of them are oriented Gabor-like edge detectors and the rest are colour blobs. Nobody designed that. It falls out of gradient descent on natural images, and it matches what neurophysiologists found in cat visual cortex in 1959. The first layer of a CNN is close to a fixed fact about the statistics of the visual world; everything above it is where architectures differ.

Stride, padding, dilation

Three integers control the geometry of a convolutional layer, and all three change the output size, the compute cost, and the receptive field in different ways. They are worth separating carefully because architectures use them for genuinely different purposes.

Stride is how far the window moves between evaluations. Stride 2 halves the output resolution and quarters the number of output positions, which quarters the FLOPs of that layer and of every layer above it. Stride is the primary downsampling instrument in modern networks, and it is free in parameters.

Padding adds a border of (usually) zeros so the window can be centred on edge pixels. With p = (k−1)/2 and stride 1, output size equals input size — the "same" convolution. Without padding, you lose k−1 pixels per layer, which is fine once and catastrophic thirty times: a 224px input run through 16 unpadded 3×3 layers ends up at 192px, and the corners of the image have been seen by fewer filters than the centre.

Dilation inserts gaps between the sampled positions, so a 3×3 kernel with dilation 2 covers a 5×5 footprint with 9 weights. It buys receptive field without buying parameters or resolution loss. This is the trick that made dense prediction — segmentation, depth — practical without a decoder.

Figure 03 · Geometry explorer

Kernel size, stride, padding, dilation

Set the four knobs and watch the window sweep. The output grid is built cell by cell, the shape formula updates live, and the cost bar shows what each setting does to MACs.

Grey cells are padding. The amber grid is the output; the mint frame is the effective footprint of the kernel including dilation gaps.

The stride/pooling equivalence, and why it stopped being one

For a long time, downsampling meant "stride-1 convolution followed by 2×2 max pool". Later architectures fold the downsample into the convolution itself with stride 2, which is cheaper — you never compute the positions you were going to discard. But they are not identical: max pooling introduces a small amount of local translation invariance and a non-linearity, while strided convolution is a pure subsample and will alias. ResNet's original stride-2 placement in the 1×1 shortcut branch throws away 3/4 of the feature map with no smoothing at all, which is exactly the bug the "ResNet-D" tweak fixed by moving the stride to the 3×3.

Receptive fields

The receptive field of a unit is the region of the input that can influence it. For a stack of convolutions it grows according to a simple recursion, where j is the cumulative stride ("jump") of the layer:

j₀ = 1, r₀ = 1 j_n = j_n-1 · s_n r_n = r_n-1 + (k_n - 1) · d_n · j_n-1 stride multiplies the jump; kernel size and dilation add to the radius, scaled by however far apart the previous layer's samples already were.

Two consequences fall straight out. With stride 1 everywhere, receptive field grows linearly in depth: n layers of 3×3 give 2n+1. With stride 2 downsampling every so often, the jump term compounds and growth becomes roughly exponential in the number of downsampling stages. This is why every architecture in this article downsamples: not to save compute (though it does), but because linear receptive-field growth would need hundreds of layers to see a whole object.

The second consequence is the argument that made VGG. Three stacked 3×3 convolutions have the same 7×7 receptive field as a single 7×7 convolution, but use 3·(3²C²) = 27C² parameters instead of 7²C² = 49C² — 45% fewer — and interleave two extra non-linearities. Deeper and thinner beats shallow and fat, as long as you can train it.

Figure 04 · Receptive field growth

How far back can this unit see?

The influence cone traced from one output unit down to the input, for four different stacking strategies. Watch the footprint expand layer by layer, and compare the parameter cost of each route to the same receptive field.

Effective receptive field is smaller than the theoretical one in practice — contributions decay roughly Gaussian toward the edges of the cone, because there are far more paths through the centre.

Why convolution wins

A fully connected layer on a 224×224×3 image with 1000 hidden units has 150 million parameters, and it has to learn separately that a cat in the top-left is a cat and a cat in the bottom-right is a cat. Convolution hard-codes three priors instead:

Locality: Pixels far apart are conditionally independent given what's between them. A unit only looks at a k×k neighbourhood.
Weight sharing: The same feature is worth detecting everywhere, so one set of weights is reused at every position. This is where the parameter reduction comes from.
Translation equivariance: Shift the input, and the feature map shifts identically: f(shift(x)) = shift(f(x)). Pooling and global average pooling then convert equivariance into invariance where you want it.

These are strong assumptions, and they are strong in the technical sense: they make the hypothesis class smaller, so you need less data to constrain it. That is exactly why CNNs dominated when datasets were 10⁶ images, and why Vision Transformers — which discard locality and sharing almost entirely — only overtook them when datasets reached 10⁸–10⁹. The prior is a loan against data: you repay it when you have enough.

1D, 2D, 3D

Nothing about the operation is inherently two-dimensional. The kernel slides over whatever axes you declare to be spatial, and sums over the axis you declare to be channels. The three common cases differ only in how many axes get swept:

Variant	Input	Kernel	Swept axes	Parameters	MACs	Typical use
Conv1D	L × C_in	k	1	k·C_in·C_out	k·C_in·C_out·L	audio, ECG, text, time series
Conv2D	H × W × C_in	k × k	2	k²·C_in·C_out	k²·C_in·C_out·HW	images
Conv3D	T × H × W × C_in	k_t × k × k	3	k_t·k²·C_in·C_out	k_t·k²·C_in·C_out·THW	video, CT/MRI volumes

The confusion people usually have is that a 2D convolution on an RGB image already involves a three-dimensional kernel — 3×3×3 — so what makes 3D convolution different? The answer is that the channel axis is summed over completely and immediately, while a spatial axis is swept. A 2D kernel sees all channels at once and produces one number per position; a 3D kernel slides along time as well, so it can learn that "edge moves left" is a different feature from "edge moves right". You cannot express motion with a 2D kernel over stacked frames-as-channels, because the sum over channels is order-agnostic once the weights are learned per slot — you get a fixed-length, non-shift-equivariant template rather than a motion detector.

Figure 06 · One operator, three sweeps

From a line, to a plane, to a volume

The same dot-product-and-slide, shown with one, two and three swept axes. The channel axis is drawn folded flat to make the distinction between "summed" and "swept" visible.

Cost scales with the product of swept extents. Moving a 3×3×64→64 layer from 2D at 56² to 3D at 16 frames multiplies MACs by 48 before you change anything else. This is why video models spent a decade avoiding full 3D convolution.

The convolution family

By about 2016 the interesting design space had stopped being "what kernel size" and become "which parts of the k×k×C_in×C_out tensor do we actually need". Every efficient architecture since is built from some factorisation of that tensor. Here is the family, with costs, using a common setting of H=W=56, C_in=C_out=128, k=3:

Operator	Parameter formula	Params	MACs	What it gives up
Standard 3×3	k²·C_in·C_out	147,456	462.4 M	—
1×1 (pointwise)	C_in·C_out	16,384	51.4 M	all spatial context
Depthwise 3×3	k²·C_in	1,152	3.6 M	all cross-channel mixing
Depthwise separable	k²·C_in + C_in·C_out	17,536	55.0 M	joint spatial-channel filtering
Grouped 3×3, g=32	k²·C_in·C_out/g	4,608	14.5 M	mixing across groups
Dilated 3×3, d=2	k²·C_in·C_out	147,456	462.4 M	sampling density (gridding)
Transposed 3×3, s=2	k²·C_in·C_out	147,456	1,849.7 M	nothing; it upsamples (and checkerboards)

The depthwise-separable row is the one that changed the industry. It costs 1/C_out + 1/k² of a standard convolution — about 8.4× less here — and loses surprisingly little accuracy, because the assumption it encodes (spatial correlation and cross-channel correlation can be filtered separately) is approximately true in learned features. Every mobile architecture from MobileNet onward is built on it, and so is Xception, and so is the ConvNeXt block.

Figure 07 · Factorising the weight tensor

Seven ways to connect C_in channels to C_out

Each stage animates one operator: which weights exist, how the sum is taken, and what it costs relative to a standard convolution. The cost bars are computed from the formulas above.

Watch the connectivity matrix on the right — it is the clearest way to see that grouped convolution is block-diagonal, depthwise is fully diagonal, and 1×1 is dense-but-pointwise.

Part twoThe lineage

The lineage

What follows is the main sequence: the architectures each of which changed a specific thing, and each of which is still legible in what people build today. Before the deep dives, here is the whole field at once. The vertical axis is ImageNet top-1 accuracy, the horizontal is compute, and the area of each mark is parameter count — three quantities that architectures have traded against each other for a decade.

Figure 08 · The accuracy–compute frontier

A decade of the frontier, plotted honestly

Play to sweep forward through time and watch the Pareto frontier move down and to the left. Hover or focus any point for its full spec; switch the horizontal axis between compute and parameters to see how differently the field looks under each.

Accuracies are single-crop, single-model, ImageNet-1k validation, as reported in the original papers. Later re-trainings with modern recipes lift almost every one of these numbers by 1–3 points, which is itself an important caveat: a chunk of the apparent architectural progress from 2015–2020 was training-recipe progress.

Layer-by-layer explorer

This is the reference instrument for the rest of the article. Pick an architecture and it is constructed live, layer by layer, from its real spec: kernel sizes, strides, channel widths, groups. For each layer you get the tensor shape, the parameter count, the MAC count, and the activation memory — the three quantities that determine whether a model fits on your hardware and how fast it runs.

Figure 09 · Architecture explorer

Every layer of nine networks, computed live

Play walks the network from input to logits. The spine shows spatial resolution collapsing and channel width growing; the two cost tracks below show where parameters and FLOPs actually sit — which is almost never where you'd guess.

Totals are computed from the layer specs in this page, not looked up. They land within ~1% of the published figures for all nine, which is the sanity check that the specs are right.

Read the two cost tracks against each other

On AlexNet and VGG, the parameter track is a cliff at the very end — the fully connected layers hold 96% and 89% of the weights respectively. The FLOP track is the opposite shape, front-loaded into the early high-resolution convolutions. On ResNet-50 and everything after it, global average pooling deletes the parameter cliff entirely, and the FLOP track flattens into a roughly uniform profile across stages. That flattening is not an accident: it is the direct consequence of the rule "halve the resolution, double the channels", which keeps MACs per stage approximately constant.

LeNet-5, 1998

LeNet-5 is 61,706 parameters and about 0.4 million MACs, and it contains, in miniature, every structural idea the next twenty years would elaborate: local receptive fields, shared weights, subsampling, and a classifier head. It read handwritten digits on cheques at scale — a deployed commercial system, not a benchmark result.

Its design choices are recognisably archaic in exactly two places. The subsampling layers are trainable average-pool-plus-scale-plus-bias with a sigmoid, not max pooling. And the C3 layer uses a hand-designed sparse connection table — each of the 16 output maps looks at a hand-picked subset of the 6 input maps — explicitly to break symmetry and to save computation. That table is, in retrospect, grouped convolution invented eighteen years early for exactly the same reason it was reinvented in AlexNet: the hardware couldn't afford dense connectivity.

What it established

Convolution → subsample → convolution → subsample → dense, the template still in use.
End-to-end gradient training of the feature extractor, rather than hand-engineered features plus a learned classifier.
Channel width grows as resolution shrinks (1 → 6 → 16 while 32 → 14 → 5).

What blocked it scaling

Sigmoid/tanh activations saturate; gradients die past a handful of layers.
No normalisation, so deeper stacks were numerically unstable.
No GPU, no ImageNet, no dropout, no augmentation pipeline. The idea was right and the substrate was missing for fourteen years.

AlexNet, 2012

AlexNet is LeNet scaled up by roughly 1000× in parameters and 1800× in compute, plus four changes that made that scaling trainable: ReLU, dropout, aggressive data augmentation, and two GPUs. It won ILSVRC-2012 with 15.3% top-5 error against 26.2% for the runner-up, a margin large enough that the field reorganised around it within a year.

The architecture itself is a compromise with 3 GB of GDDR per GTX 580. Convolutions 2, 4 and 5 are split into two groups that never talk to each other, because each half lived on a different card. That grouping is why the parameter count is 61M rather than 62.4M, and why the MAC count is 0.72G rather than 1.14G — the paper's authors described it as a memory workaround and were mildly surprised that it also improved accuracy. Fifteen years later that same trick, rebranded as cardinality, is the core of ResNeXt.

Figure 10 · Stem design

11×11 stride 4, against three stacked 3×3

Two routes to a similar receptive field, animated side by side: the aggressive single-shot downsample of AlexNet versus VGG's patient stack. Parameter cost, MAC cost and what each one actually resolves are tracked as the animation runs.

The 11×11 s4 stem discards 15/16 of the spatial positions in a single step. Zeiler and Fergus's visualisations in 2013 showed the resulting filters were a mess of dead units and aliasing artefacts; ZFNet's fix was simply 7×7 stride 2, worth about a point and a half of top-5.

Advantages

ReLU: no saturation, ~6× faster convergence than tanh at equal error.
Dropout on the two 4096-wide FC layers, without which the 58M-parameter head memorises the training set.
Overlapping max pooling (3×3, stride 2) — slightly better than non-overlapping, and it was the last architecture where anyone bothered.

Disadvantages

96% of parameters are in the classifier head; the convolutional trunk is only 2.5M.
Local response normalisation, which was later shown to do essentially nothing and was dropped everywhere.
Ad hoc, non-repeating layer specification — every layer has different hyperparameters chosen by hand. Nothing about it composes.

VGG, 2014

VGG's contribution is methodological rather than mechanical: it replaced hand-tuned per-layer hyperparameters with a single repeating motif. Every convolution is 3×3, stride 1, padding 1. Every downsample is 2×2 max pool. Channel width doubles at every downsample: 64, 128, 256, 512, 512. The only free variable is depth, and the paper is an ablation over depth — 11, 13, 16, 19 layers.

This is the first architecture you can describe in one sentence, and that regularity is why it dominated as a backbone for detection and segmentation for years after better classifiers existed. It is also ruinously expensive. VGG-16 is 138M parameters and 15.5 GMACs — 3.8× the compute of ResNet-50 for 4.5 points less top-1 accuracy — and 123M of those parameters are in the three fully connected layers, with fc6 alone holding 102M.

The activation memory nobody warns you about

VGG-16 is unusually painful to train not because of its weights but because of its activations. The first two 64-channel layers at 224×224 alone store 3.2M values per image per layer; the whole forward pass holds about 15M activations, and every one of them must be kept alive until the backward pass consumes it. At batch 32 in fp32 that's roughly 1.9 GB of activations against 0.55 GB of weights, before optimiser state. This ratio — activations dominating weights during training, weights dominating during inference — is the single most under-appreciated asymmetry in CNN engineering.

Advantages

One motif, repeated. Trivially scalable in depth, trivially transferable.
Small kernels: three 3×3 beat one 7×7 on parameters (27C² vs 49C²) and add two non-linearities.
Extremely strong transfer features; the standard perceptual-loss backbone to this day.

Disadvantages

15.5 GMACs and 138M parameters — the worst accuracy-per-FLOP of any architecture here.
Needed staged initialisation (train VGG-11, use it to init VGG-16) because plain deep stacks don't converge from scratch. This is the problem ResNet solved.
19 layers was the ceiling. Deeper VGG got worse, and not from overfitting.

Inception, 2014

GoogLeNet arrived the same year as VGG and took the opposite position: instead of picking one kernel size and repeating it, run several in parallel and let the network weight them. An Inception module computes 1×1, 3×3, 5×5 and pooled branches over the same input and concatenates the results along the channel axis.

Done naively this is unaffordable — a 5×5 convolution from 480 channels to 128 costs 1.5M parameters in a single branch. The actual contribution is the 1×1 bottleneck: project 480 channels down to 32, then do the 5×5, then let concatenation restore the width. Cost falls by roughly 10× and accuracy does not. That idea — cheap linear projection to reduce the channel dimension before expensive spatial filtering — is the ancestor of the ResNet bottleneck, the MobileNet inverted residual, and the ConvNeXt MLP block.

Figure 11 · The bottleneck trick

Naive Inception, then the version that shipped

The four parallel branches are built one at a time with running cost, then the 1×1 reductions are inserted and the cost bars collapse. Both versions produce the same output shape.

GoogLeNet is 7.0M parameters and 1.6 GMACs at 22 layers — roughly 9× fewer weights than AlexNet, with 6.7% top-5 error against AlexNet's 15.3%. It also replaced the FC head with global average pooling, which is where most of that parameter saving comes from.

Later versions kept factorising. Inception-v2/v3 replace the 5×5 with two stacked 3×3, then replace each n×n with an n×1 followed by a 1×n — a rank-1 spatial factorisation that saves another 33% — and introduce batch normalisation, label smoothing, and the auxiliary-classifier-as-regulariser reinterpretation. Inception-v4 and Inception-ResNet then fold in residual connections. The family's weakness was always that the modules are hand-designed, irregular, and awkward to modify: there is no single number you can turn to make an Inception network bigger.

Advantages

Multi-scale processing inside one layer, rather than across layers.
1×1 bottlenecks: the cheapest possible way to control channel dimensionality.
Global average pooling instead of FC — removes ~50M parameters and acts as a structural regulariser.
Best accuracy-per-parameter of its generation by a wide margin.

Disadvantages

Highly irregular; dozens of hand-tuned channel widths with no principle behind them.
Concatenation makes memory layout and kernel fusion harder — poor arithmetic intensity on real hardware, so wall-clock speed lags the FLOP count.
Auxiliary classifiers added training complexity for a benefit later shown to be small.

ResNet, 2015

The observation that motivates ResNet is a negative result: a plain 56-layer network has higher training error than a plain 20-layer one. Not test error — training error. That rules out overfitting and points at optimisation. Deeper networks should be able to represent everything shallower ones can (set the extra layers to identity), so if they can't even fit the data, the problem is that identity is hard to learn by stacking non-linear layers.

The fix is embarrassingly small. Make the layers learn a residual and add the input back:

y = F(x, W) + x If the optimal function is close to identity, F only has to learn a small perturbation — and F = 0 is trivially reachable by driving the weights to zero. The gradient path is what matters most: ∂L/∂x = ∂L/∂y · (1 + ∂F/∂x) The "1" is an ungated highway. Gradient reaches every layer at full strength regardless of depth, so the multiplicative decay through many Jacobians that kills plain networks simply doesn't happen.

Figure 12 · Gradient flow

Plain stack versus residual stack, backward pass

A gradient pulse is injected at the loss and traced backwards through 30 layers. On the left it is attenuated by each Jacobian; on the right the identity path carries it intact. The signal strength meter is on a log scale.

The residual path also changes the loss surface. Empirically the landscape of a plain deep network is chaotic and non-convex at every scale; adding skips smooths it into something with a recognisable basin, which is the geometric reading of why it trains.

ResNet-50 and above use a bottleneck block: 1×1 down to C/4, 3×3 at C/4, 1×1 back up to C. The expensive spatial convolution runs at a quarter of the width, so a 3-layer bottleneck costs about the same as a 2-layer basic block while being deeper. This is Inception's 1×1 trick, absorbed into a regular repeating unit.

Advantages

Depth stops being a liability: 152 layers trains, and 1000 layers trains with pre-activation ordering.
Regular, composable stages — one number (blocks per stage) scales the whole family from 18 to 152.
Batch normalisation everywhere; no more careful staged initialisation.
Still the default backbone for detection and segmentation a decade later.

Disadvantages

Residual addition forces every block in a stage to the same width, which constrains design.
The original block puts ReLU after the addition, so the identity path isn't truly clean; pre-activation ResNet-v2 fixes this and trains deeper.
Skip connections increase peak activation memory — you must hold x alive while F(x) is computed.
Downsampling in the 1×1 shortcut with stride 2 discards 75% of the input with no anti-aliasing.

DenseNet and ResNeXt, 2016–17

Two divergent answers to "what should the block do", both published within months of each other.

DenseNet: concatenate instead of add

Every layer in a dense block receives the concatenated feature maps of all preceding layers in that block, and contributes g new channels (the growth rate, typically 32). Layer ℓ therefore has C₀ + ℓ·g input channels and there are ℓ(ℓ+1)/2 connections in a block of depth ℓ.

Addition merges information destructively — after y = F(x) + x you cannot recover x. Concatenation preserves it, so features computed early remain directly available to every later layer, and the network never has to spend capacity re-deriving them. The result is remarkable parameter efficiency: DenseNet-121 reaches 74.4% top-1 with 8.0M parameters, and DenseNet-201 beats ResNet-50 — 77.4% against 76.1% — with 20M parameters instead of 25.6M.

The catch is memory, and it is severe. Naive implementations materialise every concatenated tensor, giving quadratic activation growth in block depth. The standard fix is shared-storage checkpointing, which trades ~15–20% extra compute for linear memory. DenseNet also has poor arithmetic intensity — lots of small operations on wide, concatenated tensors — so its wall-clock throughput badly underperforms its FLOP count on GPUs.

ResNeXt: split, transform, merge

ResNeXt takes the ResNet bottleneck and replaces the single 3×3 convolution with 32 parallel ones of width 4, summed. That is exactly a grouped convolution with g=32, and it is exactly AlexNet's two-GPU hack generalised. The new hyperparameter is cardinality, and the paper's central claim is that increasing cardinality is a more effective use of a fixed FLOP budget than increasing depth or width. ResNeXt-50 32×4d: 25.0M parameters, 4.2 GMACs, +1.7 top-1 over ResNet-50 at essentially identical cost.

Figure 13 · Two block topologies

Add-and-forget versus concatenate-and-keep

A residual stage and a dense block built side by side. Watch the channel bookkeeping: the residual stage holds width constant while the dense block grows linearly, and the connection count grows quadratically.

Both topologies solve the same optimisation problem — giving gradients a short path to early layers — but they make opposite memory/parameter trades. ResNet is parameter-hungry and memory-lean; DenseNet is the reverse.

MobileNet and SENet, 2017–19

MobileNetV1 is one idea applied uniformly: replace every 3×3 convolution with a depthwise 3×3 followed by a pointwise 1×1. Cost drops by 1/C_out + 1/9, roughly 8–9× at typical widths, for about 1 point of top-1. Two global multipliers — width α and resolution ρ — let you dial the model down to whatever latency budget you have.

MobileNetV2 adds the inverted residual with linear bottleneck, and it is genuinely counterintuitive. A ResNet bottleneck goes wide → narrow → wide and puts the skip connection between the wide tensors. MobileNetV2 goes narrow → wide → narrow and puts the skip between the narrow ones. The reasoning: the expensive depthwise filtering needs a high-dimensional space to be expressive, but the information that has to persist between blocks is low-dimensional, so route the skip through the thin tensor and save memory. And critically, drop the ReLU on the final projection — a non-linearity applied in a low-dimensional space destroys information it cannot recover, which is what "linear bottleneck" means.

MobileNetV3 then adds squeeze-and-excitation, the h-swish activation, and a hardware-aware NAS search over the block configuration, plus a hand-redesigned head after the search produced something expensive at the last stage.

Squeeze-and-excitation

SENet, which won the final ILSVRC in 2017, adds a channel attention module that any architecture can absorb: global average pool to get one number per channel, two FC layers with a reduction ratio of 16, sigmoid, then multiply each channel by its gate. It is dynamic, input-dependent channel reweighting, and it costs almost nothing — about 2.5M extra parameters on ResNet-50 and under 1% extra FLOPs — for roughly +1.5 top-1. It is also the first mechanism in this lineage that is genuinely not a convolution: it is global, and it is multiplicative.

Figure 14 · The efficient block

Separable, inverted, squeezed

Built in four movements: standard convolution, its depthwise-separable factorisation, the inverted residual that reroutes the skip, and the SE gate that reweights channels from global context.

Depthwise convolutions have terrible arithmetic intensity — one MAC per weight read — so they are memory-bandwidth-bound, not compute-bound. MobileNetV2 has 13× fewer FLOPs than ResNet-50 but is nowhere near 13× faster on a GPU. On a phone's NPU the gap is much closer to the FLOP ratio, which is the hardware it was designed for.

EfficientNet, 2019

Before EfficientNet, scaling a network up meant picking one axis and pushing it: more layers (ResNet-152), more channels (WideResNet), or bigger inputs. The paper's observation is that these axes are not independent — a deeper network needs more channels to feed it and higher resolution to have anything left to see — and that scaling them together with a fixed ratio is strictly better.

depth d = α^φ width w = β^φ resolution r = γ^φ subject to α · β² · γ² ≈ 2, α, β, γ ≥ 1 FLOPs scale as d·w²·r², so the constraint keeps the compute cost at ≈2^φ. Grid search on the B0 baseline gives α=1.2, β=1.1, γ=1.15.

The baseline B0 itself came from a NAS over MobileNetV2-style MBConv blocks with an explicit accuracy-and-latency objective. B0 is 5.3M parameters and 0.39 GMACs at 77.1% top-1; B7 is 66M and 37 GMACs at 84.3%, which at the time matched the best available model with 8.4× fewer parameters.

Figure 15 · Compound scaling

Three knobs, one coefficient

Drag φ and watch depth, width and resolution move together. The single-axis comparison shows what you get for the same FLOP budget spent entirely on one dimension — and why each of those curves saturates.

The honest caveat: EfficientNet's FLOP efficiency does not translate into training-time efficiency. Large activation footprints from high input resolutions and depthwise convolutions make B5–B7 slow to train per step. EfficientNetV2 fixed this by using ordinary fused convolutions in the early stages and by scaling resolution progressively during training.

ConvNeXt, 2022

By 2021 Vision Transformers had overtaken CNNs on ImageNet and the narrative was that attention had won. ConvNeXt is the control experiment for that claim: start from ResNet-50, apply the modern training recipe first, then apply Transformer-inspired architectural changes one at a time, and report the accuracy after each step. The result is a clean decomposition of where the gains actually came from.

Step	Change	Top-1	Δ
0	ResNet-50, original 2015 recipe	76.1	—
1	Modern recipe: 300 epochs, AdamW, RandAug, Mixup, CutMix, stochastic depth	78.8	+2.7
2	Stage ratio 3:4:6:3 → 3:3:9:3	79.4	+0.6
3	Patchify stem: 7×7 s2 + pool → 4×4 s4	79.5	+0.1
4	Depthwise conv, width 64 → 96	80.5	+1.0
5	Inverted bottleneck (MLP ratio 4)	80.6	+0.1
6	Kernel 3×3 → 7×7, moved before the expansion	80.6	+0.0
7	ReLU → GELU, fewer activations, fewer norms	81.3	+0.7
8	BatchNorm → LayerNorm	81.5	+0.2
9	Separate downsampling layers	82.0	+0.5

Read the Δ column carefully. The largest single contribution — 2.7 points — is the training recipe, applied to an architecture that was six years old and unchanged. That should recalibrate how you read every architecture comparison published between 2016 and 2021, most of which held the recipe fixed at whatever the baseline used.

Figure 16 · The modernisation ladder

ResNet-50 → ConvNeXt-T, one change at a time

Each rung is one modification. The block diagram morphs as you advance and the accuracy bar tracks the running total; the recipe step is coloured differently because it changes no weights at all.

ConvNeXt-T lands at 28.6M parameters, 4.5 GMACs, 82.1% top-1 — versus Swin-T's 28M, 4.5 GMACs, 81.3%. Matched compute, matched parameters, no attention. The conclusion is not that attention is useless; it is that a large share of what was attributed to attention was scale, data augmentation, optimiser and schedule.

Part threeComparison

One image, nine networks

Here is the rig. One image enters at 224×224 and is pushed through every architecture's real spatial schedule — the actual kernel sizes, strides and downsampling points from the specs used in Figure 09 — with a bank of oriented edge, blob and texture filters standing in for learned weights. What you are watching is not what a trained network computes; it is what each architecture's geometry does to a signal. That turns out to be the more instructive thing to see.

Figure 17 · The signature instrument

Push one image through every architecture

Choose an image and an architecture, then play. Feature maps are convolved live, stage by stage, at the true resolutions each network uses. Compare mode runs two architectures on the same input, locked to the same stage, so you can see exactly where their schedules diverge.

Upload your own image if you like — it never leaves the page; the file is read locally and drawn straight to a canvas.

The thing to watch for is when each network throws resolution away. AlexNet is at 27×27 after two layers. VGG is still at 112×112 after four. ResNet-50 drops to 56×56 in the stem and then holds it for a whole stage. ConvNeXt patchifies to 56×56 in one 4×4 stride-4 step and never looks at fine detail again. Those choices determine what the network can possibly represent, before a single weight is learned — a network that has downsampled to 7×7 cannot localise anything to better than 32 pixels, no matter how good its features are.

Time and space

Architecture papers report parameters and FLOPs. Neither is what you actually care about. Here is the full accounting, split by regime, because training and inference have qualitatively different bottlenecks.

Inference

per image, per layer: time ≈ max( MACs / peak_throughput , bytes_moved / bandwidth ) memory ≈ weights + max over the layer graph of (live activations) arithmetic intensity I = MACs / bytes_moved [MAC per byte] I high → compute bound (dense 3x3 with wide channels) I low → bandwidth bound (depthwise, 1x1 at low width, elementwise ops)

At inference you only need one layer's activations alive at a time (plus whatever skip connections hold open), so activation memory is small and the weights dominate. Peak memory for ResNet-50 inference at batch 1 is roughly 100 MB of weights against a few MB of activations. This is why quantisation — int8, or 4-bit — is the dominant inference optimisation: it attacks the term that actually dominates.

Training

per step: time ≈ 3 x forward FLOPs (1 forward + 2 backward, roughly) memory ≈ weights Ψ·P + gradients Ψ·P + optimiser state 2Ψ·P (Adam: m and v) + ALL activations kept for backward ← usually the largest term Activation term scales with batch size; the others do not. Activation checkpointing trades ~30% extra compute for O(sqrt(L)) memory.

The reversal is total. At inference the weights dominate; at training with any reasonable batch size the stored activations dominate, often by 3–5×. Every layer's output must be retained from the moment it is computed until its gradient is consumed, because the backward pass needs it to compute weight gradients. This is why an architecture with cheap weights and expensive activations — DenseNet, EfficientNet-B7 — can be miserable to train despite looking efficient on paper.

Figure 18 · Memory over one training step

Watch the activation stack fill and drain

Forward pass pushes activations onto the stack; backward pass pops them. The peak is at the boundary. Toggle activation checkpointing to see the sawtooth it replaces the plateau with, and the recomputation it costs.

Numbers are computed from the same layer specs as Figure 09, at fp32 with Adam, so they are the honest arithmetic rather than an illustration. Mixed precision roughly halves the activation term and leaves the optimiser state in fp32.

What actually mattered

Stepping back from twenty-four years, a small number of ideas did most of the work, and a surprising number of celebrated ones did not survive.

Survived: ReLU-family activations · batch/layer normalisation · residual connections · 1×1 bottlenecks · global average pooling instead of FC heads · depthwise separable convolution · repeating, parameterised stages · squeeze-and-excitation · strong augmentation and long schedules
Did not: Local response normalisation · overlapping pooling · auxiliary classifiers · hand-designed sparse connection tables · dropout in convolutional trunks (replaced by stochastic depth) · the fully connected classifier head · naive multi-branch modules with hand-tuned widths
Still contested: Max pooling versus strided convolution · kernel size beyond 7×7 · attention hybrids versus pure convolution · NAS-discovered versus hand-designed topologies · whether FLOPs mean anything for latency (they mostly don't)

The through-line is that almost every durable idea made optimisation easier rather than making the model class richer. Residual connections don't add representational power — a plain network can express anything a residual one can. Normalisation doesn't either. What they do is make the loss surface navigable, and that turned out to be the binding constraint for most of the decade. The architectures that won were the ones that were easiest to train, not the ones that were most expressive.

The second through-line is that regularity beats cleverness. VGG's one-motif design, ResNet's uniform stages and RegNet's explicit search over design spaces rather than individual models all point the same way: an architecture you can describe with four numbers is one you can scale, search, port and deploy. Inception was more accurate per parameter than its contemporaries and lost anyway, because nobody could modify it.

Where this leaves convolution in 2026

Convolution didn't lose to attention; it got absorbed. Every serious vision-language backbone has convolutional patch embedding at the front, most efficient ViT variants use depthwise convolution in the FFN for local mixing, and hybrid designs — convolutional stem and early stages, attention at low resolution — remain the practical optimum for anything latency-sensitive. The inductive bias is still worth what it costs whenever you have less than a few hundred million images. What changed is that convolution is now a component rather than the whole design.

References

Grouped by topic. Every entry links to its primary source.

The operation

One multiply-accumulate at a time

What a filter sees

Edit the nine numbers, watch the concept change

The learned version is not so different

Stride, padding, dilation

Kernel size, stride, padding, dilation

The stride/pooling equivalence, and why it stopped being one

Receptive fields

How far back can this unit see?

Why convolution wins

The same layer, two hypothesis classes

1D, 2D, 3D

From a line, to a plane, to a volume

The convolution family

Seven ways to connect Cin channels to Cout

The lineage

A decade of the frontier, plotted honestly

Layer-by-layer explorer

Every layer of nine networks, computed live

Read the two cost tracks against each other

LeNet-5, 1998

What it established

What blocked it scaling

AlexNet, 2012

11×11 stride 4, against three stacked 3×3

Advantages

Disadvantages

VGG, 2014

The activation memory nobody warns you about

Advantages

Disadvantages

Inception, 2014

Naive Inception, then the version that shipped

Advantages

Disadvantages

ResNet, 2015

Plain stack versus residual stack, backward pass

Advantages

Disadvantages

DenseNet and ResNeXt, 2016–17

DenseNet: concatenate instead of add

ResNeXt: split, transform, merge

Add-and-forget versus concatenate-and-keep

MobileNet and SENet, 2017–19

Squeeze-and-excitation

Separable, inverted, squeezed

EfficientNet, 2019

Three knobs, one coefficient

ConvNeXt, 2022

ResNet-50 → ConvNeXt-T, one change at a time

One image, nine networks

Push one image through every architecture

Time and space

Inference

Training

Watch the activation stack fill and drain

What actually mattered

Where this leaves convolution in 2026

References

Foundations

The classical lineage

Depth, residuals and reuse

Efficiency and scaling

Dilation, upsampling and dense prediction

Video and 3D convolution

Modernisation and what came after

Seven ways to connect C_in channels to C_out