A field guide · Multimodal architectures

The VLM Architecture Gallery

Seventy-one vision–language models drawn to a single standardized plate, so that the only things that move between two diagrams are the things the papers actually changed.

Vision-language modelsArchitectureConnectorsToken budgets
Anirudh SrinivasanPublished Last modified 34 min read

How do you get an image into a language model without wrecking the language model? Seventy-one architectures answer that, from Flamingo's gated cross-attention to encoders that dissolve into the base model entirely.

Every model here is a set of answers to the same five questions, and the answers interact. Choose native resolution and you inherit a variable-length sequence, which forces packing, which turns cross-rank load balancing into a real engineering problem. Choose a fixed 256-token budget and the KV cache becomes predictable enough to serve at scale, but a dense page of text arrives as mush. Nothing here is free.

The through-line is one number: how many tokens does a single picture cost? It governs serving cost, context pressure, and how many images fit in a conversation. Watch it in every figure below.

71
architectures
2021–26
five years of releases
12
connector families
33 / 52
encoder / LLM configs

Four ways to join two towers

Before the detail: the four shapes almost every model reduces to. An image becomes patches, the patches become a shorter sequence, and that sequence meets the text somewhere. Where it meets, and how much shorter it got, is the entire design space.

Figure 1The optical bench

Five decisions, and everything else is data

Strip away the naming and every architecture in the gallery is a set of answers to the same five questions. Each explainer below is an animated figure in its own right — step through the alternatives and watch what happens to the tensors.

How to read a diagram

Every architecture in the gallery is drawn to the same skeleton, so the diagrams compare by superposition: put two side by side and the only things that move are the things the paper changed. Blocks tinted coral are exactly those changes.

Preprocessing & patchesResize, tile, crop, or pass native. This row decides how many patches exist before anything is learned.
Vision transformerA dashed group with a ×N multiplier — norm, self-attention, norm, FFN, residual junctions drawn as ⊕. The footer carries width, positional scheme, and whether it stayed frozen.
ConnectorOne box per stage of the projection chain, with the compression ratio underneath. This is where token count is actually decided.
Language modelThe same dashed group, with attention type, head split, positional encoding, and either a dense FFN or an MoE row. Cross-attention and vision-expert models get an extra row inside the block — that is the whole difference.
Head & outputFinal norm, LM head with its projection width, and what the model is allowed to emit — text, boxes, points, markdown, image tokens.
What this paper changedCoral fill with a dot in the corner, wherever it sits on the plate. Read two diagrams side by side and the coral blocks are the entire difference between them.

Here is the plate assembling itself one phase at a time, with the camera tracking whichever stage is under discussion. This is the figure to internalise; everything else in the gallery is this diagram with different labels.

Figure 2The standard plate, assembled

Backbone numbers come from one shared table of encoder and LLM configurations, so two models on the same backbone are guaranteed to draw identically, and correcting one config propagates everywhere. Where a paper does not publish a figure the row reads — rather than a guess.

What a picture costs

The single number that governs serving cost, KV cache, and how many images fit in context. Log scale, coloured by connector family — watch the field split in two around 2024, when tiling arrived and one camp chose fidelity while the other chose a fixed budget.

Figure 3Visual tokens per image, 2021–2026

Hover a point for the model; click to open its plate. The vertical spread after 2024 is the whole story — two viable answers to the same question, and no convergence since.

Which lines kept which ideas

Families with more than one release, on a real time axis. Node colour is the connector in use at that release, so an abandoned idea shows up as a colour change part-way down a line.

Figure 4Family lines, coloured by connector

IDEFICS abandoning the resampler, and the OCR line turning the encoder into a compressor, both read straight off the colour changes.

Compare two architectures

Two plates, same skeleton, differing rows highlighted. This is the payoff of standardising the diagram.

Search, filter by input handling, connector, or fusion point, and open any card for the fully expanded plate, the fact sheet, and the concepts that model introduced.

References

Everything the gallery is built from, grouped by the problem each line of work was solving. Every entry links out to the report, the blog post, or the repository; model entries also open their plate here.