An Image Is a Sentence: How Vision Transformers See

An image is a sentence

A convolution makes three promises before it sees any data. Nearby pixels are related. The same edge detector is useful in the top-left corner and the bottom-right one. Meaning is built hierarchically, from edges to textures to parts to objects. Those promises are inductive biases: locality, translation equivariance, hierarchy. They are also constraints, and for a decade they were worth paying for, because they let a network learn from a hundred thousand images instead of a hundred million.

The vision transformer removes all three. It cuts the image into a grid of non-overlapping squares, flattens each one into a vector, adds a learned position code, and feeds the resulting sequence to a stack of standard transformer encoder blocks. There is no convolution after the first layer. There is no notion that patch 37 sits above patch 51 except whatever the position embeddings happen to encode. Every patch can look at every other patch in layer one.

This should not work. It works spectacularly, but only past a data threshold, and the entire subsequent literature can be read as a set of answers to two questions: how do we get the inductive biases back without giving up the flexibility, and how do we stop paying quadratic cost in the number of patches. Swin answers the second with windows. PVT answers it with key downsampling. MViT answers it with pooling. CvT and CoAtNet answer the first with convolutions bolted back in. ConViT answers it with a gate that starts convolutional and learns its way out. DeiT answers it with a teacher.

This article walks all of them. Every architecture gets a diagram you can step through frame by frame, an exact FLOP and memory accounting, and a place in three interactive labs where you can push the same image through eight different backbones and watch what each one actually does to it.

Notation used throughout

H×W input resolution · P patch size · N = HW/P² tokens · D embedding width · L depth · h heads · d = D/h head width · M window side (Swin) · R spatial-reduction ratio (PVT) · s pooling stride (MViT) · T frames, t tubelet depth (video). FLOPs are counted as multiply-accumulates ×2 unless stated. Memory is activation memory in a single forward pass, per sample, in elements.

Patchify: the only new layer

Strip a ViT down and there is exactly one component that is not lifted verbatim from a language model: the patch embedding. Everything else — pre-LayerNorm, multi-head self-attention, residual streams, a 4× MLP with GELU — is the 2017 encoder block, unchanged.

The patch embedding takes x ∈ ℝ^(H×W×C) and produces N = HW/P² vectors of width D. Conceptually it reshapes the image into N flattened patches of dimension P²C and multiplies by a learned matrix E ∈ ℝ^(P²C × D). In code it is a single strided convolution:

nn.Conv2d(in_channels=3, out_channels=D, kernel_size=P, stride=P)
# kernel == stride ⇒ non-overlapping ⇒ exactly the patch-and-project operation

That equivalence matters more than it looks. It means the "no convolutions" claim is a claim about everything after token zero. It also means the very first design knob in the entire family is whether that convolution's kernel equals its stride. Set kernel > stride and patches overlap: you have reintroduced local smoothing and, empirically, much better early-training stability. That is precisely what CvT and the "early convolutions help" line of work do.

Step through the figure below. The camera pushes in on the grid, lifts one patch out, flattens it, projects it, and drops it into the sequence beside its position embedding. Stage 7 shows the whole tensor at once, with shapes annotated.

What to watch: the flattening order is row-major and arbitrary — the model has no idea patch 15 is directly below patch 1 until the position embedding tells it. Stage 5 is where the entire 2D structure of the image is thrown away and handed back as a learned lookup table.

The class token, and where it comes from

ViT prepends one extra learned vector, x_class, to the sequence. It carries no image content. After L blocks, its final state is passed to the classification head. The idea is inherited straight from BERT's [CLS]: give the model a register with no job except aggregation, and let attention decide what to write into it. The sequence the encoder actually sees is length N+1.

The quadratic wall

Self-attention over N tokens computes an N×N score matrix. Two consequences follow, and they are different consequences.

Time. Forming QKᵀ costs 2N²D FLOPs and applying the weights to V costs another 2N²D. The projections cost 8ND² (four matrices: Q, K, V, O). The MLP with expansion 4 costs 16ND². So one block is

FLOPs_block = 24·N·D² + 4·N²·D

The crossover — where attention stops being a rounding error and starts dominating — is at N = 6D. For ViT-B (D=768) that is N = 4608 tokens, or roughly a 1088×1088 image at patch 16. Below that resolution, a ViT is mostly a stack of matrix multiplies and the quadratic term is a minority of the bill. Above it, the picture inverts fast.

Memory. This is the harsher constraint, and it binds long before the FLOP crossover. A naive attention implementation materialises the h·N² score matrix, and during training it must keep it for the backward pass. At 224² with patch 16 that is 12 heads × 197² ≈ 466k floats per layer per sample — trivial. At 1024² it is 12 × 4097² ≈ 201M floats per layer per sample, or 400 MB in fp16, times 24 layers. FlashAttention removes the N² materialisation by tiling and recomputing, dropping activation memory for attention to O(N) and leaving only the O(N²) time. Every complexity claim in this article distinguishes the two.

What to watch: the token grid grows linearly with area while the score matrix grows with area squared. At stage 5 the projection term and the attention term cross; past it, no amount of width reduction saves you — only fewer tokens or a sparser attention pattern does.

The three escape routes

Everything that follows takes one of three exits from this wall. Reduce N over depth — a pyramid, as in PVT, Swin, MViT. Restrict who attends to whom — windows, as in Swin; axial or divided attention, as in TimeSformer. Shrink the keys, not the queries — spatial-reduction attention in PVT, pooling attention in MViT. They are not mutually exclusive; Swin does the first two, MViT does the first and third.

Anatomy of a ViT block

The block is pre-norm, which is the detail that makes deep ViTs trainable at all. Both sublayers normalise their input and add their output to an untouched residual stream:

z'_ℓ = MSA( LN(z_{ℓ-1}) ) + z_{ℓ-1}
z_ℓ = MLP( LN(z'_ℓ) ) + z'_ℓ

Post-norm — the original 2017 arrangement — puts LayerNorm after the residual add, which means the identity path is repeatedly rescaled and the gradient at initialisation decays with depth. Pre-norm leaves a clean additive highway from the loss to layer one. Every ViT variant in this article is pre-norm.

The MLP is two linear layers with a GELU in between, expanding D → 4D → D. It holds 8D² parameters against attention's 4D², so two thirds of a ViT's weights live in the MLPs and only one third in attention. That ratio is worth remembering when you read parameter counts.

What to watch: the residual stream (the thick horizontal band) is never normalised and never scaled. Each sublayer reads a normalised copy and writes an increment back. This is why you can splice extra tokens in — a distillation token, registers, text tokens — without destabilising anything.

Inside one attention head

Take a 4×4 patch grid, so 16 tokens. One head projects each token to a query, a key and a value of width d. The score between token i and token j is q_i·k_j / √d; softmax over j turns row i into a distribution; the output for i is that distribution's weighted sum of values.

Attn(Q,K,V) = softmax( QKᵀ / √d ) V
Q,K,V ∈ ℝ^(N×d) · scores ∈ ℝ^(N×N) · output ∈ ℝ^(N×d)

The √d matters: without it the dot products have variance d, the softmax saturates, and gradients vanish. With it, variance is 1 at initialisation and the distribution starts near-uniform — which is exactly why a fresh ViT attends almost globally in every layer and has to learn locality.

Multi-head simply runs h of these in parallel at width d = D/h and concatenates. The FLOP count is identical to one head at width D for the projections, but the score matrices are h separate N×N objects, so multi-head is strictly more expensive in attention memory. Heads are the unit of specialisation: in a trained ViT you can find heads that attend to a fixed spatial offset, heads that pool globally, and heads that only ever look at the class token.

What to watch: the highlighted query row is a distribution over the whole grid, not over a neighbourhood. Nothing in the operation knows the grid is 2D. Stage 6 shows the same weights redrawn spatially, which is the only place 2D structure appears — and only because we chose to draw it that way.

Position: four answers

Self-attention is permutation-equivariant. Shuffle the tokens and the outputs shuffle identically. For language that is a bug fixed by position embeddings; for vision it is the same bug, with an extra wrinkle — position is two-dimensional, and the resolution changes between pretraining and fine-tuning.

Four families of answer appear across this literature:

Learned 1D absolute. Original ViT. A table of N+1 vectors added to the token embeddings. Simple, effective, and the reason ViT needs bicubic interpolation of the position table whenever you change input resolution.
Fixed 2D sinusoidal. Half the channels encode row, half encode column. Extrapolates to unseen resolutions for free but is empirically a wash with learned 1D at fixed resolution.
Relative position bias. Swin's choice. A learned scalar B_{Δi,Δj} added to the score before softmax, indexed by the relative offset between two tokens inside a window. Because it is relative, it is translation-equivariant by construction, and because a window is M×M, the table has only (2M−1)² entries.
Rotary, extended to 2D. Rotate query and key subspaces by an angle proportional to row and column index. The dot product then depends only on the offset. Now standard in modern VLM vision towers because it interpolates cleanly across resolution.

The bias-versus-embedding distinction is the important one. An additive embedding perturbs the content of a token — a position-carrying vector flows through the value path and gets mixed into the output. An additive score bias perturbs only the routing. Swin's relative bias never touches the values.

What to watch: stages 5 and 6 show what happens at fine-tuning time when you move from 224 to 384. The learned table must be reshaped to a 14×14 grid, bicubically resized to 24×24, and re-flattened — a step that is easy to get silently wrong, because forgetting to hold out the class-token row shifts every position by one.

CLS token or global pool

ViT reads out from the class token. Most hierarchical variants — Swin, PVT, MViT — use global average pooling over the final feature map instead. The choice is not cosmetic.

A class token is a learned query. It has its own row in every attention matrix and can implement a content-dependent, non-uniform pooling: "attend hard to whatever looks like an object, ignore background". Global average pooling is a fixed uniform pooling, which throws away that flexibility but costs one fewer token, removes an awkward special case from every windowed or pooled attention scheme, and produces a feature map that dense-prediction heads can consume directly.

That last point is decisive for the pyramid architectures. A detection head wants a C4/C5 feature map at H/16 × W/16. A class token is not a feature map. Any architecture aiming at detection and segmentation drops the class token, which is why the hierarchical branch of this family tree is also the GAP branch.

What to watch: the class token's attention weights (stage 3) are peaked on the object; the average pool's implicit weights (stage 4) are flat by construction. Stage 6 shows why the pyramid variants need the flat version anyway.

The missing inductive bias

The clearest way to see what a ViT gives up is to measure mean attention distance: for each head, average the pixel distance between a query patch and the patches it attends to, weighted by attention. A convolution's equivalent is its receptive field, which is fixed by architecture and grows linearly with depth.

Trained ViTs show a distinctive pattern. In the earliest layers, some heads have very small mean attention distance — they have learned something convolution-like — while other heads in the very same layer already attend across the entire image. By the middle of the network, essentially all heads are global. A ResNet cannot do the second thing at layer 2 no matter how it is trained; the information is not there yet.

This is the trade in one picture. The ViT can route information anywhere at any depth, which is why it scales so well with data; the CNN cannot, which is why it learns so well without much data. When ViT is trained on ImageNet-1k alone it underperforms a comparable ResNet. Pretrained on JFT-300M it overtakes everything. The crossover sits somewhere around 10–30M images.

What to watch: the vertical spread inside each ViT layer is the point. Early layers contain both near-local and fully-global heads simultaneously. The CNN's grey staircase has no spread at all — every unit in a layer has the same receptive field.

DeiT and the data problem

DeiT's claim was that ViT does not actually need 300M images — it needs the right recipe. The recipe had two halves.

The first half was augmentation and regularisation borrowed wholesale from the CNN world and turned up: RandAugment, Mixup, CutMix, random erasing, stochastic depth, repeated augmentation, and a long cosine schedule. This alone closed most of the gap on ImageNet-1k.

The second half was distillation through attention. Add a second special token, the distillation token, alongside the class token. It participates in self-attention exactly like every other token, so it can develop its own view of the image. At the output, the class token is supervised by the ground-truth label and the distillation token is supervised by the teacher's prediction. At inference the two heads' logits are averaged.

L = ½·CE( ψ(z_cls), y ) + ½·CE( ψ(z_dist), y_teacher )
hard-label variant: y_teacher = argmax of the teacher's logits

The counter-intuitive result: the best teacher is a convolutional one, not a bigger transformer. A RegNet teacher transfers its inductive bias to the student through the label stream. The two tokens also end up genuinely different — their final embeddings have cosine similarity around 0.06, and they disagree on a meaningful fraction of images, which is why averaging them helps.

What to watch: the distillation token is not a head bolted on at the end — it is present from layer one and attends to image patches throughout. That is what "through attention" means, and it is why it outperforms a plain KL term on the class token's logits.

PVT: the pyramid returns

ViT has one resolution and one width for its entire depth. Every CNN backbone since 2014 has four stages at 1/4, 1/8, 1/16, 1/32 with widths that double as resolution halves. PVT asks the obvious question: why not both?

The architecture is four stages. Each begins with a patch-embedding layer that halves spatial resolution and increases width, then applies some number of transformer blocks. Stage 1 starts at H/4 × W/4, which for a 224 input is 56×56 = 3136 tokens. Plain global attention there would cost 4 × 3136² × 64 ≈ 2.5 GFLOPs per layer, which is already more than an entire ResNet-18 forward pass.

The fix is spatial-reduction attention (SRA). Queries stay at full resolution — you need one output token per input token. Keys and values are reshaped back to a 2D map, downsampled by a factor R with a strided linear projection, and flattened again:

K, V ← Reshape(x, R) · W_R with Reshape: ℝ^(N×D) → ℝ^(N/R² × R²D)
SRA(Q,K,V) = softmax( Q Kᵀ / √d ) V scores are now N × N/R²

PVT uses R = 8, 4, 2, 1 for the four stages. The attention cost becomes 4N²D/R². At stage 1 that is a 64× reduction. The keys have not been thrown away — they have been averaged into N/R² summary tokens, so every query still sees the entire image, just at coarser granularity. That is a genuinely different bargain from Swin's, which keeps full key detail but restricts the query's reach.

What to watch: the score matrix stops being square. It is N × N/R², which is why SRA is sometimes described as cross-attention from a fine grid onto a coarse one. PVT-v2 later replaces the strided linear with average pooling plus a depthwise convolution, making R effectively adaptive and removing the fixed-resolution assumption.

Swin: windows, and the shift

Swin is the most-copied design in this family and the one most worth understanding mechanically, because the trick is not "local attention" — that is easy — it is how you get global information flow anyway, without any cross-window operator.

The base move: partition the h×w token map into non-overlapping M×M windows (M=7 throughout the paper) and run full self-attention inside each window independently. Cost drops from quadratic in hw to linear in hw. But now window boundaries are hard walls. A token at the edge of window A can never influence window B, at any depth.

The fix, and the paper's whole contribution: in every second block, shift the entire window grid by ⌊M/2⌋ pixels diagonally before partitioning. Windows in the shifted layer straddle the boundaries of windows in the unshifted layer, so information crosses. Two consecutive blocks are therefore not identical — they are a W-MSA block followed by an SW-MSA block, and Swin's depths are always even for exactly this reason.

ẑ_ℓ = W-MSA( LN(z_{ℓ-1}) ) + z_{ℓ-1}
z_ℓ = MLP( LN(ẑ_ℓ) ) + ẑ_ℓ
ẑ_ℓ₊₁ = SW-MSA( LN(z_ℓ) ) + z_ℓ
z_ℓ₊₁ = MLP( LN(ẑ_ℓ₊₁) ) + ẑ_ℓ₊₁

The shift creates a problem it then has to solve. Shifting a 8×8 grid of windows by 3 produces 9×9 windows, most of them partial — different sizes, ragged batching, wasted compute. Swin avoids this with cyclic shift plus attention masking: roll the feature map torus-style so the number of windows is unchanged, then, because some windows now contain tokens that were not spatially adjacent before the roll, add a mask of −∞ to the score matrix wherever two tokens came from different original regions. Softmax zeroes them. After attention, roll back.

This is the single most-implemented-wrong part of Swin. The mask is not optional decorum — without it, the bottom-right window attends across the image's wrap-around seam and the model learns a toroidal image.

What to watch: stages 4–7 are the cyclic-shift trick. Notice that after the roll, one window contains four differently-coloured regions — those are four originally-distant patches now sharing a window. The mask at stage 6 is what stops them from talking. Stage 9 is patch merging: 2×2 neighbouring tokens concatenated to 4D and linearly projected to 2D, which is the pyramid's downsampling step.

Patch merging is not pooling

Between stages, Swin concatenates each 2×2 group of tokens along the channel axis — 4D — and applies a linear layer to 2D. Compare with average pooling, which would map 4D → D by discarding three quarters of the information, or with a strided convolution, which mixes spatially and across channels at once. Merging is lossless up to the linear layer's rank: nothing is thrown away before the projection gets to choose. This is the same operation as PixelUnshuffle followed by a 1×1 convolution.

Deriving Swin's complexity

This derivation is worth doing explicitly because the constant factors decide whether the design is actually a win at realistic sizes.

Let the token map be h×w with width D. Global MSA:

Ω(MSA) = 4·h·w·D² + 2·(hw)²·D
projections (Q,K,V,O) scores + weighted sum

Windowed MSA over hw/M² windows of M² tokens each. The projection term is unchanged — every token is still projected exactly once. The attention term is per-window and there are hw/M² windows:

Ω(W-MSA) = 4·h·w·D² + 2·(hw/M²)·(M²)²·D
= 4·h·w·D² + 2·M²·h·w·D

The quadratic term in hw has become linear, with constant M². At Swin's stage 1 — h=w=56, D=96, M=7:

Term	Global MSA	Window MSA (M=7)	Ratio
Projections `4hwD²`	115.6 MFLOP	115.6 MFLOP	1.0×
Attention	`2(hw)²D` = 1 887 MFLOP	`2M²hwD` = 29.5 MFLOP	64×
Block total	2 003 MFLOP	145 MFLOP	13.8×
Score memory / head	3136² = 9.83 M	64 × 49² = 0.15 M	64×

Two things to take from the table. First, the memory saving is the same 64× as the FLOP saving on the attention term, and unlike FlashAttention's saving it costs no recomputation. Second, the block-level speedup is only 13.8×, not 64×, because the projection term is untouched and quickly dominates. Windowing solves the attention problem and then hands you back a network whose cost is dominated by dense matmuls — which is exactly where hardware is happiest.

At stage 4 — h=w=7, D=768 — the window is the whole feature map, W-MSA and MSA are identical, and the attention term is 2·49²·768 = 3.7 MFLOP against 4·49·768² = 115.6 MFLOP of projections. The quadratic term is 3% of the block. Swin's design spends its complexity budget exactly where the tokens are.

CvT, ConViT, CoAtNet

Three different theories of how to put convolution back, arranged from most surgical to most structural.

CvT — convolutions inside the transformer

CvT makes two changes and drops position embeddings entirely.

Convolutional token embedding. Between stages, instead of non-overlapping patchify, use an overlapping strided convolution (kernel 7 stride 4, then kernel 3 stride 2). Overlap means adjacent tokens share input pixels, which restores local continuity and gives a smooth resolution pyramid.

Convolutional projection. Replace the linear W_Q, W_K, W_V with depthwise-separable convolutions applied to the 2D token map. For keys and values, use stride 2 — which is both a locality prior and a 4× reduction of the attention matrix, PVT-style, for free.

Because a depthwise convolution is inherently position-sensitive, the network knows where things are without any position embedding. Removing them is not a simplification for its own sake: it means CvT handles arbitrary input resolutions with no interpolation step at all.

ConViT — a gate that starts convolutional

ConViT's gated positional self-attention is the most elegant idea in this section. Each head computes two attention maps: a normal content-based one, and a purely positional one that depends only on the relative offset r_{ij} between tokens. A learned per-head gate σ(λ) mixes them:

A_ij = (1−σ(λ)) · softmax( q_i·k_j ) + σ(λ) · softmax( v_pos · r_ij )

Initialise v_pos so the positional term reproduces a specific convolutional kernel, and initialise the gate hard towards positional. At step zero the head is a convolution. As training proceeds, heads that benefit from content-based routing open the gate and become attention; heads that do not stay convolutional. The inductive bias is a soft prior with an escape hatch rather than a hard architectural constraint. On small data it behaves like a CNN; on large data it behaves like a ViT; nobody has to choose in advance.

CoAtNet — convolution stages then attention stages

CoAtNet's argument is about generalisation versus capacity. Depthwise convolution has better generalisation (a strong, translation-equivariant prior); self-attention has better capacity (input-adaptive, global). It unifies them by observing that a depthwise convolution and relative attention are both sums over a neighbourhood with a weight that is either static or dynamic — and then simply adds the static term inside the softmax:

y_i = Σ_j ( exp(q_i·k_j + w_{i−j}) / Σ_k exp(q_i·k_k + w_{i−k}) ) · x_j

Here w_{i−j} is a learned scalar depending only on relative position — the convolution kernel — and it lives inside the softmax so it is translation-equivariant and adaptive at once. The macro design then places convolutional blocks in the high-resolution early stages (where attention is unaffordable) and relative-attention blocks in the low-resolution late stages: the C-C-T-T layout that a systematic search picked over C-T-T-T and C-C-C-T.

What to watch: stages 4–6 sweep ConViT's gate from fully positional to fully content-based on a live attention map. This is the clearest single picture of what an inductive bias is: a starting point in weight space, not a wall.

MViT: pooling attention

MViT's framing is different again. Rather than restricting attention or downsampling keys with a projection, it pools all three of Q, K, V with a learned pooling operator, and controls the output resolution through the query stride alone.

PoolAttn(X) = softmax( P_Q(Q) · P_K(K)ᵀ / √d ) · P_V(V)
P is a depthwise conv with stride s; stride_Q sets output resolution, stride_K/V set the key budget

When stride_Q = 2, the output has a quarter the tokens of the input — the block itself performs the downsampling, so no separate patch-merging layer is needed. When stride_Q = 1 and stride_K = stride_V = s, the resolution is preserved but the attention matrix is N × N/s², which is PVT's bargain reached by a different route.

MViT's other structural idea is the channel–resolution scaling rule inherited from CNNs: whenever resolution drops 4×, width doubles, keeping per-stage compute roughly balanced. It starts deliberately tiny — D=96, a single head — and ends at D=768 with 8 heads. A ViT is a rectangle; MViT is a wedge.

MViTv2 adds two corrections that matter in practice: decomposed relative position embeddings (a separable R_h + R_w bias rather than a full 2D table, cutting the parameter count from O(hw) to O(h+w)), and a residual pooling connection that adds the pooled query back to the attention output, restoring information the pooling destroyed.

What to watch: the query pool and the key pool are independent knobs. Stage 5 holds the query stride at 1 and raises the key stride, which is exactly PVT's SRA; stage 6 raises the query stride, which is downsampling. Same operator, two different jobs.

Video: tubelets

Extending to video means extending the tokenisation. Two options.

Uniform frame sampling. Take T frames, patchify each independently exactly as an image ViT would, concatenate. You get T · N tokens and the model must learn all temporal structure through attention. Simple; no new parameters.

Tubelet embedding. Extend the patch into time: a t × P × P spatio-temporal volume, embedded by a 3D convolution with kernel and stride (t,P,P). You get (T/t) · N tokens, and each token has already fused t frames of motion at the input. Fewer tokens and a temporal inductive bias for the price of a 3D conv.

The token counts are brutal either way. Sixteen frames at 224² with patch 16 is 16 × 196 = 3136 tokens; joint space-time attention over that is a 3136² matrix per head per layer. Tubelets with t=2 halve it to 1568. This is why every video transformer factorises something.

What to watch: the tubelet is doing the same job in time that patchify does in space — trading resolution for sequence length before attention ever runs. Every token-count saving here is squared in the attention matrix.

TimeSformer's five schemes

TimeSformer enumerates the space of attention patterns over a space-time token grid and benchmarks all of them. This is the cleanest ablation in the video literature, and the results are counter-intuitive enough to be worth memorising.

Scheme	A query attends to	Keys per query	Attention cost
Space (S)	all patches in its own frame	`N`	`T·N²`
Joint space-time (ST)	every patch in every frame	`TN`	`(TN)²`
Divided (T+S)	same patch across time, then own frame	`T + N`	`TN·(T+N)`
Sparse local-global (L+G)	local 3D neighbourhood, then strided global	`≈ 3N/4 + TN/s²`	intermediate
Axial (T+W+H)	same row, then same column, then same time	`T + √N + √N`	lowest

Divided space-time attention wins, and it beats joint attention not only in cost but in accuracy. Two blocks of reasoning explain why. Divided attention has separate learned projections for the temporal and spatial passes — 2× the attention parameters of joint — so it has more capacity where it counts. And joint attention over 3136 tokens is a very hard optimisation problem with a very large hypothesis space; factorising is a regularising prior that says "space and time are different axes", which is true.

Cost at 8 frames, 196 patches per frame: joint attention has 1568² = 2.46M pairs per head; divided has 1568 × (8 + 196) = 0.32M. A 7.7× reduction, rising to 32× at 32 frames.

Interactive: pick a scheme, then click any patch in any frame to make it the query. Highlighted cells are the keys it reaches in one block. The readout gives keys-per-query and total pairs; switch between schemes with the same query selected to see the reach collapse.

ViViT's four factorisations

ViViT explores the same space one level higher — not "which pairs attend" but "at what granularity do we factorise the model".

Model 1 — Spatio-temporal attention. Joint attention over all (T/t)·N tokens. The baseline. Quadratic in total tokens, and the memory, not the FLOPs, is what stops you.
Model 2 — Factorised encoder. A full spatial transformer runs on each frame independently and produces one per-frame representation; a second, smaller temporal transformer runs over those T representations. Two separate encoders, late fusion. The cheapest by far, and it maps perfectly onto an image-pretrained backbone — you initialise the spatial encoder from a ViT and train only the temporal one from scratch.
Model 3 — Factorised self-attention. One encoder, but each block does spatial attention then temporal attention, with the tokens reshaped between them. This is TimeSformer's divided attention, arrived at independently. Same parameter count as Model 1, far less compute.
Model 4 — Factorised dot-product. One attention operation, but half the heads attend over space and half over time, then concatenate. Same parameter count and same FLOPs as Model 1's projections, with the attention matrices factorised. The most surgical of the four.

In accuracy order on Kinetics-400 the models are close, with Model 1 marginally best when you can afford it and Model 2 the best accuracy-per-FLOP by a wide margin. ViViT also confirmed the initialisation detail that makes all of this practical: initialise the temporal attention's output projection to zero, so the model begins life exactly equal to its image-pretrained spatial backbone and learns temporal structure as a perturbation.

What to watch: Models 3 and 4 have identical parameter counts to Model 1 but different attention shapes. Model 2 is the outlier — it is two networks, and the temporal one sees only T tokens, which is why it is roughly an order of magnitude cheaper than the rest.

Lab: one image, eight backbones

This is the figure worth spending time in. Pick an image, pick a backbone, and watch what that backbone actually does to it — stage by stage, at the real token resolutions and widths from each paper.

Three things are drawn simultaneously. The token map shows the image at the resolution that stage operates on, each cell being one token. The reach overlay shows, for the token you click, exactly which other tokens it can attend to in one block of that stage — global for ViT and DeiT, a 7×7 window for Swin, the full image at reduced key resolution for PVT, a pooled neighbourhood for MViT, a local convolutional field for CvT's early stages and CoAtNet's convolutional stages. The ledger gives per-stage tokens, width, attention pairs, FLOPs and score-matrix memory, computed live from the formulas in this article rather than quoted.

Click through the stages of Swin and then of ViT with the same query patch selected. ViT's reach is the whole image at every stage and never changes. Swin's reach is 49 tokens at every stage — but because the token map shrinks, those 49 tokens cover 1/64th of the image at stage 1 and the entire image at stage 4. That is the pyramid's real argument, and it is much easier to see than to say.

Interactive: choose an image and a backbone above, click any cell in the token map to move the query, and use the stage controls to walk the pyramid. The ledger updates live. Try: CoAtNet at stage 1 (convolutional, 3×3 reach) then stage 3 (transformer, global reach) — the architecture changes character halfway through.

Lab: the complexity bench

The formulas collected in one place, with sliders. Everything is computed from these expressions, per block, per sample:

Scheme	Attention FLOPs	Projection + MLP FLOPs	Score memory (naive)	Score memory (Flash)
Global MSA	`4N²D`	`24ND²`	`h·N²`	`O(N)`
Window MSA (M)	`4NM²D`	`24ND²`	`h·N·M²`	`O(N)`
SRA (R)	`4N²D/R²`	`24ND² + ND²/R²`	`h·N²/R²`	`O(N)`
Pooling attn (s)	`4N²D/s²`	`24ND² + conv`	`h·N²/s²`	`O(N)`
Divided space-time	`4TN·(T+N)·D`	`48TND²`	`h·TN·(T+N)`	`O(TN)`
Joint space-time	`4(TN)²D`	`24TND²`	`h·(TN)²`	`O(TN)`

A note on the divided row: it has 48TND² projections, not 24, because divided attention runs two attention sublayers per block with independent W_Q,W_K,W_V,W_O. That is where its extra parameters come from, and it is a real cost that the "7.7× cheaper" headline hides.

Interactive: drag resolution to 1024 and watch the global bar detach from the rest. Toggle Train / Infer to switch between activation-memory accounting (train: all activations retained for backward) and peak-working-set accounting (infer: one block live at a time). Toggle Flash to remove the N² materialisation and see how much of the memory problem is an implementation artefact rather than an architectural one.

Lab: reach and receptive field

One block's reach is a local property. What matters for whether an architecture can relate the top-left corner to the bottom-right one is effective receptive field growth over depth, and the schemes differ enormously.

Global attention reaches everything at depth 1. Windowed attention without shifting never reaches outside its window at any depth. Windowed attention with shifting expands its reach by roughly M/2 tokens per block pair, so covering a 56×56 stage-1 map takes about 16 blocks — which Swin does not have at stage 1, and does not need, because patch merging shrinks the map faster than reach needs to grow. Axial attention reaches everything in exactly 2 blocks (one row pass, one column pass) but with a very restricted path.

What to watch: unshifted windows plateau immediately and stay flat forever — the pathology Swin's shift exists to fix. Shifted windows and the 3×3 convolution both grow, but the window grows in jumps of M/2 against the convolution's steady 1 per layer, which is a ~3.5× faster expansion at M=7.

Training cost vs inference cost

These are different problems and the architectures rank differently on them.

Training

Backward costs roughly 2× forward in FLOPs, so total training compute is about 3× the forward pass, or 4× with activation checkpointing (forward, recompute-forward, backward). The binding constraint is almost never FLOPs — it is activation memory, because every intermediate needed by the backward pass must be retained across the whole depth.

Per block, per sample, ignoring attention scores, a ViT retains roughly: the block input ND, the LN output ND, Q/K/V 3ND, attention output ND, the MLP hidden state 4ND, plus the GELU's input 4ND. That is about 13ND elements, times L layers. For ViT-L at 224² — N=197, D=1024, L=24 — that is 63M elements per sample, 126 MB in fp16, before attention scores. Add naive attention scores at h·N²·L = 16 × 38809 × 24 = 14.9M and it is a small addition; at 1024² input it becomes 100× larger than everything else combined. This is the single strongest practical argument for FlashAttention in high-resolution ViTs, and it is why the memory ranking between architectures collapses once Flash is on.

The second training-specific cost is one this literature rarely mentions but every practitioner hits: for a classifier with a large label space or a VLM head with a large vocabulary, the logits tensor can dwarf the backbone. B × N × V in fp32 for the softmax is a real tensor. Chunked or fused cross-entropy is not optional at that scale.

Inference

No activations are retained; peak memory is one block's working set plus weights. The ranking is now dominated by arithmetic intensity and how well the operator maps to the hardware.

Plain ViT is the fastest per FLOP. It is a stack of large, regular GEMMs with no reshaping, no masking, no gather. It hits close to peak on every accelerator and is trivially compiled and quantised.
Swin's window partition and cyclic shift are memory-bound reshapes. They contribute no FLOPs and measurable latency. A Swin block at equal FLOPs is typically slower in wall-clock than a ViT block. The mask adds an N×M² add before every softmax.
PVT's SRA and MViT's pooling add real work — a strided projection or depthwise convolution per block — which the FLOP formula counts, but they also break the clean GEMM pattern.
CoAtNet's convolutional stages are extremely hardware-friendly, and its transformer stages run at low resolution where attention is cheap. It is consistently near the top of accuracy-per-latency charts for this reason rather than for any FLOP advantage.

There is no KV cache in any of this. These are encoders: one forward pass over a fixed-length sequence, no autoregressive decode. That removes the dominant inference-memory concern of language models entirely, and it is why vision towers are so often the cheap part of a VLM despite looking expensive on paper.

The compare rig

Everything above, side by side, on one clock. Four architectures run simultaneously on the same input; the timeline is normalised so you can watch where each one spends its depth, where resolution drops, and where the attention pattern changes character.

Interactive: swap any of the four slots with the selectors, then scrub the depth playhead. The lower panel plots cumulative FLOPs against normalised depth — the shape of that curve, not its endpoint, is what distinguishes a rectangle from a wedge.

The design-choice matrix

Model	Tokenise	Pyramid	Attention	Position	Readout	Attn complexity
ViT	P=16 non-overlap	none	global	learned 1D abs	CLS	`O(N²D)`
DeiT	P=16 non-overlap	none	global	learned 1D abs	CLS + dist	`O(N²D)`
PVT	overlap-free, 4 stages	4/8/16/32	SRA, R=8/4/2/1	learned, per stage	GAP	`O(N²D/R²)`
Swin	P=4 then merge	4/8/16/32	window M=7 + shift	relative bias	GAP	`O(NM²D)`
CvT	overlapping conv	4/8/16	conv-projection, K/V stride 2	none	CLS (last stage)	`O(N²D/4)`
ConViT	P=16 non-overlap	none	gated positional SA	relative, gated	CLS	`O(N²D)`
CoAtNet	conv stem	2/4/8/16/32	conv ×2 then rel-attn ×2	relative, in-softmax	GAP	`O(N²D)` at low res
MViT	overlapping conv	4/8/16/32	pooling attention	decomposed relative	CLS	`O(N²D/s²)`
TimeSformer	frame patches	none	divided time then space	learned space + time	CLS	`O(TN(T+N)D)`
ViViT-2	tubelet t×P×P	none	spatial enc. then temporal enc.	learned space + time	CLS ×2	`O(TN²D + T²D)`

Advantages and disadvantages, honestly

Model	Strengths	Weaknesses
ViT	Simplest possible design; scales best with data and parameters; pure GEMMs so highest hardware efficiency per FLOP; trivially extended with extra tokens; the default vision tower for VLMs.	Data-hungry below ~30M images; quadratic in tokens so high resolution is punishing; single-scale output is bad for detection and segmentation; needs position-table interpolation to change resolution.
DeiT	ImageNet-1k-only training reaches competitive accuracy; distillation token is a genuinely better mechanism than logit KL; recipe transfers to every later model.	Requires a strong pretrained teacher; adds a token and a head; no architectural change so all of ViT's complexity issues remain.
PVT	First convolution-free pyramid; drops into detection and segmentation heads directly; SRA keeps global reach at every stage.	Fixed `R` per stage assumes a fixed input resolution; key downsampling loses fine detail exactly where stage 1 should have it; v1's strided linear is parameter-heavy.
Swin	Linear in image size; strong dense-prediction results; relative bias is translation-equivariant; window size is a clean accuracy/cost dial.	Cyclic shift and masking are fiddly and memory-bound; reach grows slowly with depth; window size is tied to feature-map size so changing resolution is not free; slower per FLOP than ViT.
CvT	No position embeddings at all, so arbitrary input resolution works out of the box; overlapping embedding stabilises early training; strided conv projection gets key reduction for free.	Depthwise convolutions are memory-bound; the convolutional prior costs some of ViT's large-data scaling; fewer downstream ecosystems built on it.
ConViT	Best small-data behaviour per parameter in this list; the gate is interpretable and you can read off which heads stayed convolutional; no cost at inference over ViT.	Extra positional attention map costs training compute; benefit shrinks to nothing at large data scale; single-scale like ViT.
CoAtNet	Excellent accuracy-per-latency across the whole size range; relative attention inside the softmax is principled; conv stages absorb the high-resolution cost.	Two operator families to tune and deploy; stage layout found by search so less transparent; attention only in the last two stages limits early global routing.
MViT	Block-level downsampling removes a separate merging layer; excellent on video; wedge shape balances per-stage compute well; v2's decomposed position embeddings are cheap.	Pooling destroys information (hence v2's residual pooling fix); many hyperparameters; pooling convolutions break the GEMM pattern.
TimeSformer	Divided attention is cheaper and more accurate than joint; scales to long clips; simple to implement on top of any ViT.	Doubles attention parameters per block; no spatial pyramid so dense video tasks are awkward; still quadratic in `N` within a frame.
ViViT	Model 2 is by far the best accuracy-per-FLOP; tubelets fuse motion at the input; zero-init temporal projection makes image-pretrained initialisation exact.	Model 1 is memory-prohibitive beyond short clips; Model 2's late fusion cannot model fine-grained motion within a frame; needs strong image pretraining to work at all.

How the family evolved

Read the timeline as three overlapping waves. 2020–2021, make it work: ViT proves the premise at scale, DeiT removes the data requirement, and within months PVT and Swin add the pyramid while CvT, ConViT and CoAtNet re-admit convolution. 2021–2022, make it self-supervised: BEiT, MAE, DINO and SimMIM show that the data-hunger problem has a second solution — don't label the data. MAE in particular exploits a property unique to this architecture: you can drop 75% of the tokens from the encoder entirely, because there is no fixed grid to preserve. 2022–2025, make it a component: CLIP and SigLIP turn the ViT into a text-aligned encoder, EVA and ViT-22B push scale, and NaViT, FlexiViT and Hiera attack the last rigidities — fixed resolution, fixed patch size, and the accumulated complexity of the hierarchical designs.

Hiera is the appropriate coda. It takes MViT, deletes almost all of its specialised components — the relative position embeddings, the convolutional projections, most of the pooling machinery — trains the remainder with MAE, and gets a better and faster model. The conclusion is not that the inductive biases were wrong; it is that with a strong enough pretext task the model can learn them, and hand-designed versions were leaving speed on the table.

What to watch: the lineage arrows. Almost every hierarchical model traces back to PVT or Swin, almost every hybrid to CvT, and the entire 2023-onward branch runs through MAE. Note the convergence at the right edge — the field is returning to plainer ViTs with better training, not more elaborate architectures.

Choosing, in practice

A short decision procedure, ordered by how much it actually matters.

Classification or a VLM vision tower, with pretrained weights available: plain ViT, almost always. SigLIP2 or DINOv2 weights, patch 14 or 16. The simplicity buys compilation, quantisation, sequence-parallelism and easy resolution changes, and the accuracy gap to hierarchical models has closed.
Detection or segmentation: a pyramid, or ViTDet's approach of building a pyramid on top of a plain ViT with simple deconvolutions. If you want the pyramid in the backbone, Swin and MViTv2 are the two mature choices; Swin has the larger ecosystem.
High resolution — documents, medical imaging, remote sensing: windowed or pooled attention is not optional. Also check that your attention kernel is Flash-backed before you conclude the architecture is the problem.
Small dataset, no pretraining possible: ConViT or CoAtNet, or a CNN. This is the one regime where the pure transformer is still genuinely the wrong tool.
Video: start with ViViT Model 2 (factorised encoder) initialised from an image ViT. Move to divided space-time attention only if late fusion is demonstrably losing you fine-grained motion.
Latency-critical deployment: measure, do not compute. The FLOP ranking and the wall-clock ranking disagree, and they disagree most for exactly the architectures whose FLOP counts look best.

The through-line of the last five years is that architectural inductive bias is a substitute for data, not a complement to it. Every mechanism in this article buys sample efficiency or compute efficiency by constraining what the model can express. When data is scarce, that is a good trade. When data is abundant, the constraint becomes the ceiling — which is why the field has spent 2020 to 2022 adding structure and 2022 to 2026 carefully taking it back out.

References

Grouped by topic. Every entry links to its primary source. The structure of this list follows 0xD4rky/Vision-Transformers, a well-organised zero-to-hero reading path from ViT through ViViT.

Reading paths and repositories

Zero-to-Hero: ViT · 0xD4rky · a curated ViT-to-ViViT reading path with implementations
vit-pytorch · Phil Wang · single-file reference implementations of most models in this article
timm · Ross Wightman · the canonical weights and benchmark harness for this family