One Transformer, Two Modalities: Transformers vs Vision Transformers

Take a modern language model and a Vision Transformer, delete the first layer and the last layer of each, and diff what remains. You will find almost nothing. The same residual stream, the same multi-head attention, the same two-layer MLP with a wide hidden dimension, the same layer norms in the same places. ViT was not a new architecture. It was the claim that an image can be made to look like a sentence, and that once it does, the 2017 machine works unmodified.

That claim is worth taking literally, because it tells you exactly where to look for the differences. They are all concentrated at the boundaries: how a raw input becomes a sequence of vectors, how position gets injected, which entries of the attention matrix are allowed to be non-zero, and what the last hidden state is asked to predict. The middle is shared. This article walks the whole stack, block by block, with an animated figure for every boundary.

1 · Tokenizing vs patchifying

A Transformer consumes a sequence of d-dimensional vectors. It does not care where they came from. Everything upstream of that is a front door, and the two modalities build very different doors.

Text arrives as a string of Unicode characters, which is a discrete symbolic object. The tokenizer — BPE, WordPiece, or a unigram model — is a compression algorithm fit once on a corpus and then frozen. It greedily merges frequent byte pairs until it has a vocabulary of typically 32k to 256k entries, then segments any new string into the longest pieces it knows. Crucially, this stage is not part of the network. It has no parameters that receive gradient, it is deterministic, it is discrete, and it is lossy in one specific way: two different strings can never map to the same token sequence (byte-level BPE is invertible), but the segmentation itself is an arbitrary artifact of the corpus it was fit on. " transformers" may be one token; " Transformers" may be three.

An image arrives as a dense grid of floats, already continuous, already fixed-size, with no symbols anywhere. Patchification cuts that grid into a regular tiling — 14×14 or 16×16 pixels per patch is the standard — and reads the tiles in raster order. There is no vocabulary, no merge table, no corpus statistics, nothing fit in advance. The operation is a reshape. It is differentiable, it is exactly invertible when stride equals kernel size, and it throws away nothing except the notion that neighbouring tiles were ever adjacent.

Figure 1The two front doors — from raw input to a sequence

The asymmetry that matters

A token is an index. It carries no information by itself — 4327 is not larger or smaller than 4326 in any meaningful sense, and the network can only learn what it means by looking it up. A patch is a value. Its 768 numbers already are the content; the network's job is to re-express them, not to look them up.

2 · Where the vectors come from

Both doors have to produce the same thing: an N × d matrix. But the mechanisms are structurally different, and the difference shows up in parameter count, in gradient sparsity, and in what "initialization" even means.

Text: a lookup table that is really a matmul

The embedding matrix E ∈ ℝ^(V×d) holds one learned row per vocabulary entry. Token ID i selects row i. Frameworks implement this as a gather because it is cheap, but mathematically it is onehot(i)ᵀ E — a matrix multiply with an extremely sparse left operand. That framing matters, because it explains the backward pass: the gradient with respect to E is non-zero only on rows whose tokens appeared in the batch. A 100k-vocabulary embedding table trained on a batch of 4k distinct tokens updates 4% of its rows and leaves the rest untouched. This is why embedding tables are usually kept in fp32, why they get their own sparse optimizer paths, and why rare tokens stay near their initialization for a long time.

Vision: a linear projection that is really a convolution

Each patch is flattened to P²·C numbers (16·16·3 = 768) and multiplied by a single learned matrix W ∈ ℝ^(768×d) with a bias. Every patch in every image uses the same W. This is exactly nn.Conv2d(3, d, kernel_size=16, stride=16), and most implementations write it that way because the convolution kernel fuses the reshape and the matmul. The gradient is dense: every patch of every image contributes to every element of W on every step.

Figure 2Where the numbers come from — table lookup vs linear projection

Learned or pre-computed?

This question has a clean answer in 2026 and a messier history, and it is worth separating them.

Both are learned, both from random init, both end-to-end. No modern Transformer or ViT ships with a frozen input embedding. E and W are ordinary parameters trained by the same optimizer as everything else.
Text used to be pre-computed. word2vec, GloVe and fastText vectors were trained separately and dropped in frozen, because in 2014 you could not afford to learn 300-dimensional vectors for 400k words from your own task data. Transformers killed this: pretraining is now the thing that learns the embeddings, and the "downstream task" inherits them already trained.
Vision was never pre-computed at the patch level — the projection is too cheap and too task-coupled to be worth freezing. What does get frozen in vision is one level up: an entire pretrained encoder (CLIP, SigLIP, DINOv2) is often used as a fixed feature extractor. That is a frozen network, not a frozen embedding layer.
The exception that proves the rule: generative image models (VQ-VAE, VQGAN, and the discrete branch of image generation) deliberately quantize patches against a learned codebook so images become integer IDs, and then feed those IDs through an ordinary text-style embedding table. When you want images to be causally generable, you first make them symbolic. That is a direct admission that the discrete/continuous split is the real dividing line, not text/image.

The parameter accounting is lopsided in a way people rarely notice. For a 7B-class LLM with V = 100k and d = 4096, the embedding table alone is 410M parameters — 6% of the model, and the same again at the output if weights are not tied. For a ViT-L with d = 1024, the patch projection is 768 × 1024 ≈ 0.79M parameters, which is 0.26% of the encoder. The vision front door is essentially free; the language front door is a major line item, and it is also the reason large-vocabulary models feel memory-hostile during training — the logits tensor B × N × V is the single largest activation in the graph.

3 · Assembling the sequence

Having vectors is not the same as having a batch. Text sequences are ragged and need structural markers; image sequences are rectangular by construction and need almost nothing.

Language models wrap the content in special tokens — BOS to give the first real token something to attend to, EOS to mark termination as a predictable event, plus PAD to square off the batch and, in instruction models, a whole grammar of role delimiters. Padding is pure waste: a batch of sequences with lengths 40 and 900 spends 96% of the short row computing on nothing, which is why sequence packing and variable-length attention kernels are standard now.

A ViT has no ragged-ness problem. At fixed resolution 224 with patch 16, every image is exactly 196 tokens — always, forever, no masking required. The only special token in the original design is [CLS], prepended so that one position can aggregate the whole image for classification. It is optional: many encoders instead global-average-pool the patch tokens, which works about as well and is cheaper. DINOv2 later added four register tokens — extra learned slots with no input content — after observing that the model was hijacking a few high-norm patch tokens as scratch space and corrupting their local features.

Figure 3Assembling the sequence — special tokens, padding, and length

Sequence length also scales differently. Text length is set by the data and the context window you are willing to pay for. Image length is set by geometry: N = (H/P)·(W/P), quadratic in resolution. Going from 224 to 448 does not double the sequence — it quadruples it, from 196 to 784, and attention cost with it. This single fact drives most of the design space in high-resolution vision: smaller patches, windowed attention, pooling connectors, token pruning, and the tiling schemes that document VLMs use to read dense text without going to 4096-token images.

4 · Position, the only geometry

Self-attention is permutation-equivariant. Shuffle the rows of the input and the output rows shuffle identically — the operation sees a set, not a sequence. So every ounce of spatial or sequential structure that the model uses has to be injected deliberately, and this is the block where text and vision differ most interestingly, because text has one axis and images have two.

Figure 4Position — the only place geometry enters the network

The language lineage

The 2017 paper used fixed sinusoidal encodings: each dimension pair is a sine and cosine at a different frequency, added to the embedding. Nothing is learned; the appeal was that relative offsets become linear functions, and that it extrapolates to unseen lengths in principle. GPT-2 and BERT switched to a plain learned absolute table of shape L_max × d, which is simpler and slightly better in-distribution but hard-caps the context length. Modern LLMs have converged almost entirely on RoPE: instead of adding anything to the residual stream, it rotates each 2-dimensional slice of Q and K by an angle proportional to the position. The dot product of a rotated query and a rotated key then depends only on the difference of positions, giving relative encoding for free, at every layer, with zero parameters. It is also the reason context extension works at all — you can rescale the rotation frequencies (NTK, YaRN, linear interpolation) and stretch a 4k model to 128k with light fine-tuning.

The vision lineage

The original ViT did something almost lazy: a learned 1D table over the raster-scan index, exactly as if the 196 patches were 196 words. The authors tested 2D-aware variants and reported no meaningful gain — and when you visualize the learned table's cosine similarity, you see it has recovered 2D structure on its own: the embedding for patch (3,4) is most similar to (3,5) and (4,4). The prior was learnable from data, so hand-coding it bought nothing at that scale.

It bought something later, though, once resolution stopped being fixed. A 1D absolute table trained at 14×14 is meaningless at 24×24, so every fine-tune at higher resolution reshapes the table to a grid and bicubically interpolates it — a step so standard it is a one-line utility in every ViT codebase. Newer encoders avoid the problem structurally: factorized row+column embeddings, 2D RoPE (rotate one half of the head dimension by the x index and the other half by the y index), or relative position biases per attention head as in Swin. Video and VLM stacks extend this again — M-RoPE in Qwen2-VL splits the rotation across time, height and width so that a frame index, a row and a column all live in the same positional scheme as text.

The one-line summary

Text positions are ordinal: 1D, causal, unbounded, and the model must extrapolate. Image positions are spatial: 2D, bounded, bidirectional, and the model must interpolate. Extrapolation and interpolation want different machinery, which is why the two lineages diverged even though both started from the same sinusoids.

5 · Causal vs bidirectional

This is the difference with the largest downstream consequences, and it is one boolean in the code.

A language model is trained to predict the next token, so position t must never see position t+1. The attention logits get an additive mask of −∞ above the diagonal, softmax sends those to zero, and information flows strictly left-to-right. The payoff is enormous: one forward pass over a length-N sequence yields N supervised predictions at once. The cost is that generation is inherently serial, which is what the KV cache exists to amortize — because token t's keys and values can never change when token t+1 arrives, you store them and never recompute.

A ViT is an encoder. The whole image is present at once, there is no "next patch" to protect, and the top-left corner has every reason to look at the bottom-right. The mask is simply absent: a full, dense, symmetric attention pattern where every patch sees every patch from layer one. Nothing analogous to a KV cache exists because nothing is generated incrementally.

Figure 5Who may look at whom — the mask and its consequences

The split is about the objective, not the modality, and you can see that by looking at the exceptions. BERT is text with a bidirectional mask, because masked-token prediction does not need causality. Autoregressive image generators are pixels or codes with a causal mask, because they generate in raster order. MAE is a ViT that reconstructs, still bidirectional. And a VLM uses both in one sequence: the image tokens form a bidirectional prefix, the text continuation is causal, so the mask is block-structured rather than triangular.

6 · The identical middle

Everything from here to the output head is, module for module, the same code. Project to queries, keys and values; split into heads; score; scale by 1/√d_h; mask if there is a mask; softmax; weight the values; concatenate; project out. If you handed someone the middle of a ViT-L and the middle of a 7B LLM with the shapes stripped, they could not tell you which was which.

Figure 6Multi-head attention — the part that is genuinely identical

The differences that do exist here are about scale economics, not semantics. Language models at 7B and above almost universally use grouped-query attention, sharing one KV head across several query heads, because the KV cache is the memory bottleneck at long context and high batch — a decision driven entirely by inference. Vision encoders keep full multi-head attention: there is no cache to shrink, the sequence is short, and the parameter saving is not worth the capacity loss. Similarly, sliding-window and other sparse patterns show up in LLMs to tame 128k contexts, and in ViTs only in the specific form of window attention for very high-resolution or hierarchical designs like Swin.

7 · Inside one block

The residual block is where the two families were briefly different and then converged. The 2017 Transformer used post-norm: the normalization sits after the residual addition, on the main path. It works, but the gradient at initialization is badly scaled, which is exactly why the original recipe needed a learning-rate warmup and was notoriously touchy. GPT-2 moved LayerNorm to the front of each sublayer — pre-norm — leaving an unbroken identity path from the input embedding to the final hidden state. ViT was pre-norm from birth, because it inherited the 2020 consensus rather than the 2017 one. So on this axis they are the same, and the shared answer is "pre-norm, plus a final norm before the head."

Figure 7One block, dissected — norm placement, MLP, and modern divergence

The MLP is the same shape in both: expand by 4×, apply a nonlinearity, contract. It holds roughly two-thirds of the block's parameters in both families. The stylistic divergence is in the accessories, and it maps neatly onto what each family is optimized for:

LLM-side: RMSNorm instead of LayerNorm (no mean subtraction, marginally cheaper), SwiGLU instead of GELU (three matrices instead of two, hidden dim shrunk to ~2.7× to compensate), no biases anywhere, grouped-query attention, RoPE.
ViT-side: LayerNorm with biases, GELU, LayerScale (a learned per-channel gain initialized near zero on each residual branch, so a deep ViT starts as near-identity), and stochastic depth — randomly dropping whole residual branches during training, a regularizer that matters far more in vision because the datasets are smaller relative to the model.
Both, converging again: QK-normalization. Scaling ViT to 22B and LLMs past a few hundred billion both produced attention-logit blowup, and both fixed it by normalizing queries and keys before the dot product.

8 · The exits

The last hidden state is N × d in both cases. What you attach to it is the second-largest difference in the whole comparison.

A language model applies a linear map to the vocabulary at every position, producing an N × V logits tensor, and takes cross-entropy against the input shifted by one. The output matrix is frequently tied to the input embedding table — the same parameters read and write. Because the loss is computed at every position, one sequence gives you thousands of gradient signals, which is a big part of why language pretraining is so sample-efficient per token.

A vision encoder emits one prediction per image. Take the [CLS] vector or the mean of the patch tokens, apply a small head, done. What that head is depends on the objective, and vision has many more of them than language does: supervised classification into 1000 or 21k classes; contrastive alignment against a text encoder (CLIP, SigLIP), where the "label" is the caption and the loss is over the batch's similarity matrix; masked image modelling (MAE) which reconstructs pixels for masked patches with a tiny decoder; or self-distillation (DINO, DINOv2) where a student matches an EMA teacher across augmented crops. Language has essentially one pretraining objective and vision has four live ones, because "predict the next patch" turns out to be a much worse learning signal than "predict the next word."

Figure 8The exits — language-model head vs vision objectives

9 · Inductive bias, and why ViT needed 300 million images

One consequence follows from everything above and deserves naming. A CNN hard-codes two priors: locality (a filter only sees a neighbourhood) and translation equivariance (the same filter everywhere). A ViT has neither. After patchification the model does not know that patch 3 is next to patch 4 — position embeddings are its only clue, and they are learned. Attention is global from layer one, so nothing forces early layers to be local.

That is why the original ViT underperformed ResNets on ImageNet-1k alone and only pulled ahead when pretrained on JFT-300M: with enough data, the learned prior beats the hand-coded one, but below that crossover the hand-coded one wins. DeiT then showed the gap could be closed with heavy augmentation and distillation from a CNN teacher instead of more data, and hybrid designs (convolutional stems, hierarchical stages) put a little locality back for the low-data regime.

Text has no analogous story, because language has no equivalent of translation equivariance to give up. There was never a convolutional prior in NLP worth defending, which is part of why the Transformer took over language in two years and vision in five.

10 · Cheat sheet

Every row below is a place the two architectures actually differ. Everything not in this table is shared.

Stage	Language Transformer	Vision Transformer
Input unit	Subword token (BPE / unigram)	Pixel patch, 14×14 or 16×16
Vocabulary	Finite, 32k–256k, fit on a corpus	None — continuous inputs
Front door differentiable?	No; tokenizer is frozen preprocessing	Yes; patchify is a reshape
Embedding	Row lookup in E (V×d)	Linear projection W (P²C×d) ≡ Conv2d stride P
Embedding params	Huge (e.g. 410M at V=100k, d=4096)	Tiny (≈0.8M at d=1024)
Embedding gradient	Sparse — only rows seen in the batch	Dense — every patch touches all of W
Pre-computed vectors?	Historically (GloVe); now always learned	Never at patch level; encoders frozen instead
Special tokens	BOS, EOS, PAD, role delimiters	Optional [CLS]; registers in DINOv2
Sequence length	Ragged, data-driven, 512 → 1M	Fixed, geometric, N = HW/P²
Length scaling	Linear in document length	Quadratic in resolution
Padding	Required; masked out	Not needed at fixed resolution
Position encoding	1D; RoPE dominant, extrapolated	1D learned → 2D / 2D-RoPE, interpolated
Attention mask	Causal (lower triangular)	None — fully bidirectional
Attention flavour	GQA + sliding window at scale	Full MHA; windowed only in hierarchical ViTs
Block	Pre-norm, RMSNorm, SwiGLU, no bias	Pre-norm, LayerNorm, GELU, LayerScale, DropPath
Output head	Vocab logits at every position, often tied	One vector per image → classifier or projector
Objective	Next-token cross-entropy	Supervised / contrastive / MIM / self-distillation
Inference	Autoregressive, serial, KV-cached	Single forward pass
Inductive bias	Order only	None — locality must be learned

Where they merge again

A vision-language model is the punchline to all of this. A ViT encodes the image bidirectionally into patch tokens; a connector — a linear layer, an MLP, or attentional pooling — maps those into the language model's d and, in effect, into its vocabulary space; and then the LLM treats them as ordinary tokens in its causal stream. Two front doors, one corridor. The image tokens were never symbols and never will be, but by the time they reach block one of the decoder, the machine has stopped being able to tell.

11 · References

Grouped by the part of the stack they belong to. Every row is a link to the paper.

One Transformer, Two Modalities