Inside a Modern Vision Encoder

Every argument in this guide is downstream of one question: what does the encoder actually receive? Start there. Pick an image shape, pick an intake strategy, and watch what survives the trip.

The intake bench

pick a shape → pick a strategy → watch what survives

Input image

Encoder intake strategy

source · 1024 px long side

intake

what the transformer actually receives

Visual tokens: —
Sampled at: —
Aspect distortion: —
Pixels discarded: —
Test bars resolved: —

The source is a synthetic page carrying a five-step resolution target — bar pairs at 24, 15, 9, 5 and 3 px in the master image. A bar is counted as resolved when its stroke survives at ≳1.6 output pixels after resampling. It is a proxy for "can this encoder still read 8pt type", not a benchmark.

Optics · what a vision encoder actually is

Pixels in, a sequence of vectors out. Everything else is a choice.

A vision encoder is the component that turns an image into tokens the language model can attend to. It sits before the connector, and its output is a sequence Z ∈ R^(N×D) — N spatial positions, D dimensions each. Every design argument in this field is about how you get N, what each position means, and what supervision shaped D.

Fig. 1 — patches to tokens · hover the grid, change the patch size

patch size

special tokens

The six knobs that define an encoder

Two encoders can have identical parameter counts and behave completely differently. These are the dimensions that actually vary, and every one of them leaks into downstream VLM behaviour.

Knob	Typical range	What it controls	Where it bites
Patch size p	14 · 16 · 32	Tokens = HW/p². p=14 gives 30% more tokens than p=16 at equal pixels	OCR on small type; compute per image
Input resolution	224 → 518 → native	How many pixels ever reach the model	Everything document-shaped
Position encoding	learned abs · interpolated · 2D-RoPE · M-RoPE	Whether the model can generalise to unseen shapes	Native-resolution support, video
Pooling head	CLS · attention pool (MAP) · mean	What the "image vector" means	Retrieval vs dense tasks
Training objective	contrastive · SSL · generative · distilled	What the features are about	Semantics vs geometry (see §4)
Feature layer tapped	last · penultimate · multi-layer	How abstract the tokens are	VLM quality — quietly, by 1–3 points

The one everyone trips on: most LLaVA-lineage VLMs do not use the encoder's final layer. They tap the penultimate block, because the last layer of a contrastive encoder is specialised for the pooled image-text similarity objective and throws away spatial detail on the way. If you swap encoders and forget to re-check which layer you're reading, you can lose more than the encoder swap gained.

Intake · preprocessing, the undocumented half

Nine things happen to your image before a single attention head sees it.

Papers describe the architecture. Config files describe the preprocessing. The gap between them is where most reproduction failures live — a wrong normalisation constant or a resize without antialiasing costs more accuracy than a 10× parameter increase buys.

Decode & orient

JPEG/PNG → RGB array. Alpha is flattened (usually onto white, sometimes black — they differ for screenshots and charts). EXIF orientation is applied or silently ignored depending on the library.

Gotcha · PIL applies EXIF rotation only if you ask. A phone photo pipeline that forgets it feeds the encoder sideways images, and the model will confidently describe a rotated world.
Resize — and the algorithm matters more than you'd think

Nearly all encoders specify bicubic resampling with antialiasing. Downsampling a 4000px document to 336px without an antialias filter aliases thin strokes into noise: the text is still "there", but as high-frequency garbage rather than legible glyphs.

Gotcha · OpenCV, PIL and PyTorch's interpolate historically disagreed on antialiasing defaults for the same nominal algorithm. Two "identical" preprocessing pipelines can produce measurably different scores. Match the training-time library, not just the training-time flag.
Aspect handling: squash, crop, or pad

Original CLIP resizes the short side then centre-crops — it throws away the edges of a non-square image. Many VLM pipelines instead squash to a square, distorting geometry. Letterbox padding preserves both but wastes tokens on grey bars. There is no free option; the bench above lets you feel the difference on a receipt.

Gotcha · Centre-crop on a screenshot deletes the navigation bar and the footer. Squashing a bar chart changes the apparent slope of every line. Both failure modes are invisible in aggregate benchmark numbers and obvious in production.
Snap to a patch multiple

Native-resolution encoders round height and width to multiples of the patch size (Qwen uses multiples of 28 = 14×2, because a 2×2 merge follows). SigLIP 2's NaFlex resizes so both dimensions are patch multiples while keeping distortion minimal and the sequence under a target length.
Pad to a packed sequence — and build the mask

Variable-length inputs get padded so a batch fits a tensor. Padding tokens must be masked out of attention and out of any pooling. NaViT's "Patch n' Pack" instead packs several images into one sequence with block-diagonal masks so no compute is wasted on padding at all.

Gotcha · A missing pad mask does not crash. It quietly lets grey padding participate in attention and pooling, and the loss barely notices while your dense features rot.
Normalise with the right constants

Not a detail. Encoders were trained with different channel statistics and using the wrong ones is a silent, uniform accuracy tax.

OpenAI CLIP · mean (0.481, 0.458, 0.408) std (0.269, 0.261, 0.276)
SigLIP / SigLIP 2 · mean 0.5 std 0.5 — the "half" convention
DINOv2, InternViT · ImageNet mean (0.485, 0.456, 0.406) std (0.229, 0.224, 0.225)
Patchify & project

Non-overlapping p×p blocks are flattened and pushed through one linear layer (implemented as a strided conv). This is the only place raw pixels are touched; from here everything is a token.
Add position information

Learned absolute embeddings must be interpolated to a new grid if you change resolution — bicubic interpolation of the 2D position grid, which works surprisingly well but degrades far from the training shape. 2D-RoPE (Qwen2.5-VL) and factorised M-RoPE handle arbitrary shapes natively.

Gotcha · SigLIP 2 NaFlex interpolates well between its trained sequence lengths and extrapolates poorly beyond them. "Supports any resolution" almost always means "any resolution inside the trained envelope".
Set the token budget

Native-resolution models expose min_pixels / max_pixels knobs. These are not performance settings — they decide how many tokens each image costs and therefore whether small text survives. Serving defaults are often much lower than evaluation defaults.

Gotcha · A model that scores 94 on DocVQA in a paper can score in the 70s in your service because someone capped max_pixels to control latency. Always log the realised token count per image.

Sampling · resolution is the whole ballgame

Five ways to feed a 4-megapixel document into a 224-pixel brain.

Every encoder was pretrained at some fixed shape. Real images are not that shape. The strategies below are the field's answers, in rough historical order — and the shift from "resize it in preprocessing" to "handle it inside the encoder" is the single biggest architectural change of the last two years.

2021 → · fixed square

Resize to 224 or 336 and accept the loss

One resolution, one position grid, constant cost. Fine for "is there a dog", useless for a spreadsheet. CLIP ViT-L/14@336 gives 576 tokens no matter what you feed it.

2023 → · positional interpolation

Bicubic-interpolate the position grid, then fine-tune

Cheapest way to raise resolution on an existing checkpoint. How SigLIP 2 produces its 256/384/512 checkpoints: resume from the base run at 95% completion, resize the position embeddings, keep training.

2024 → · AnyRes tiling

Cut the image into tiles, encode each, add a thumbnail

LLaVA-NeXT picks a grid that minimises aspect distortion, encodes every tile at the native 336px, and prepends a downsampled global view so the model doesn't lose the forest. Detail scales, but so do tokens — quadratically.

2024 → · tiling + compression

InternVL: dynamic tiles, then pixel shuffle

1–12 tiles of 448px chosen by closest aspect ratio, plus a thumbnail. Each tile is 1024 patches, squeezed to 256 tokens by a 2× space-to-depth shuffle before it ever reaches the LLM.

2023 → · native resolution

NaViT / Qwen: don't resize at all

Patch n' Pack packs variable-length patch sequences into one training example with block-diagonal masks. Token count becomes proportional to image area, aspect ratio is preserved exactly, and position must come from something shape-agnostic like 2D-RoPE.

2025 → · adaptive & selective

Spend tokens where the information is

InternVL3.5-Flash routes each tile to a 256- or 64-token compressor by semantic richness. PS3 scales pretraining to 4K by processing selected regions. PTP prunes by combining visual saliency with attention from the instruction tokens.

Token cost calculator

drag the dimensions — fixed-resolution rows ignore them, which is the point

Width 1024 px

Height 1024 px

Bars are scaled to the largest row. Compute in the encoder grows roughly with token count; attention inside the LLM grows with the square of total context, which is why a 20,000-token image is not "just 20× more expensive".

What tiling actually does to an image

Fig. 2 — attention scope · click a tile to see what it can and cannot see

geometry

Pixel shuffle: the compression trick hiding in every InternVL config

Tiling explodes token counts, so InternVL compresses before the LLM. Pixel shuffle (space-to-depth) takes an r×r block of spatial positions and stacks them along the channel dimension: sequence length drops by r², channel width grows by r², and no information is thrown away — it is rearranged. A 448px tile is 1024 patches at p=14; with r=2 that becomes 256 tokens. SmolVLM pushes r=3 for 9× compression. Qwen does the same job with a learned 2×2 merge in the MLP connector instead.

Fig. 3 — pixel shuffle · drag the factor, hover a block to follow it

compression factor r 2

Diminishing returns are real and they have a shape. 224 → 336 helps almost every task. 336 → 448 helps OCR and documents. Past 448 the gains become entirely task-dependent: more pixels help small text and dense charts, and do essentially nothing for "what colour is the car". Resolution should be set by your worst-case document, not maximised on principle.

Objectives · what the encoder was actually taught

Training methodology beats parameter count, and it isn't close.

The headline result across the whole literature: a 400M SigLIP 2 encoder outperforms a 5.9B InternViT-6B on most VLM benchmarks. Twenty times the parameters, losing. What separates them is the objective — what signal shaped the features, and therefore what the features are about.

Contrastive: softmax coupling vs sigmoid independence

CLIP's loss normalises over the whole batch: every image competes with every text in the same softmax. That makes the loss batch-coupled, forces enormous batch sizes (32k+) and an all-gather across every device. SigLIP reframes it as independent binary classification on each pair — a sigmoid per cell. Same matrix, no row-wise normalisation, and the gradient for a cell no longer depends on the rest of the row.

Fig. 4 — the batch similarity matrix, two ways

Diagonal = matched pairs. Off-diagonal = negatives. Hover any cell.

Why this mattered more than it sounds. Decoupling the loss made large-batch training stable and distributed training simple — but the bigger finding was that once you remove the softmax, you don't actually need the giant batch. Beyond roughly 32k, batch size stops buying accuracy. The "CLIP needs a supercomputer" constraint was partly an artefact of the loss, not a law of nature.

Self-supervised: learning geometry with no words at all

DINOv2 never sees a caption. A student network is trained to match a teacher whose weights are an exponential moving average of the student's own — no labels, no text, just consistency across crops. Three losses stack: image-level self-distillation across global views, patch-level masked prediction (iBOT), and the KoLeo regulariser, which pushes features to spread out in the embedding space and prevents collapse. DINOv3 scaled this to 7B parameters using Gram anchoring to stop dense features degrading over a long run.

Fig. 5 — self-distillation · step through one training iteration

The objective decides the failure mode

Objective	Supervision	What it makes features good at	Characteristic weakness	Seen in
Contrastive	image–text pairs	semantics, zero-shot naming, text alignment, OCR-ish reading	image-level bias; weak dense/spatial features; batch-size hunger	CLIP, SigLIP, MetaCLIP, EVA-CLIP
+ captioning decoder	image → text tokens	localisation, referring expressions, grounded description	extra decoder to train; needs box/region data	SigLIP 2 (LocCa stage), CapPa
Self-supervised	images only	segmentation, depth, correspondence, counting, geometry	no text alignment — needs bridging to enter a VLM	DINOv2/v3, iBOT, MAE, Web-SSL
Autoregressive	next-token on image+text	native compatibility with an LLM decoder; simple, scalable recipe	younger literature; fewer public checkpoints	AIMv2, OpenVision 2
LLM-aligned	LLM supervision, staged	tight coupling with a specific language model	very expensive; 20× params for marginal general gains	InternViT-6B, SAILViT
Multi-teacher distillation	frozen teachers, no labels	one backbone with CLIP semantics + DINO geometry + SAM boundaries	inherits teacher artifacts; needs careful loss balancing	AM-RADIO / C-RADIO, UNIC, Theia

What contrastive encoders see

"a dog on a beach" — instantly
Words in an image, because alt-text taught them to read
Category-level distinctions that appear in captions

What self-supervised encoders see

Which pixels belong to the same object
Relative depth and surface layout
Orientation, count and pose — things captions rarely mention

CLIP-blind pairs. "Eyes Wide Shut" found image pairs that CLIP embeds almost identically while DINOv2 separates them cleanly — different orientation, different count, different position. The MMVP benchmark built from those pairs put most MLLMs, including GPT-4V at the time, below random on embarrassingly simple visual questions. The blind spot is inherited from the encoder, not the LLM, and no amount of language-side scaling fixes it. This is the single strongest argument for mixing self-supervised features into a VLM.

Atlas · who is descended from whom

There are about a dozen families. Everything else is a variant.

Encoder names multiply faster than encoder ideas. Nearly every checkpoint in production traces back to one of the lineages below, and knowing which one tells you most of what you need: what supervision shaped it, what resolutions exist, and what it will be bad at.

Solid lines: direct lineage or initialisation. Dashed: adopted or distilled from.

Filter

Walkthrough · one image, twenty-one encoders

What actually happens between a JPEG and a token.

Same input every time: a photographed A4 page, 1200 × 1600. Step through any encoder below and watch where it diverges — the intake geometry, the patch grid, how position is encoded, which tokens can see which, where the features get tapped, and how many tokens finally reach the language model. The numbers are the real ones for each configuration. Every major family is here, including the two that never feed a language model at all (both JinaCLIPs, which fill a vector index instead) and the one that exists purely as a scaling experiment (ViT-22B at 22 billion parameters, still looking at a 224-pixel square).

Encoder

CLIP OpenCLIP Supervised ViT MetaCLIP Data-centric SigLIP JinaCLIP DINO InternViT NaViT Generative Distilled

Step 1 of 7

the pattern

Divergence happens early

By step 2 these twenty-one encoders are already looking at between 196 patches (DINO v1) and 9,804 (Qwen) of the same page — a fiftyfold spread decided entirely by intake, before a single attention head runs. Everything downstream follows from it.

the pattern

Special tokens are not cosmetic

A CLS token, an attention-pooling head and four register tokens are three different answers to "where does global information live". Get it wrong and it lives in random background patches instead.

the pattern

The architecture is rarely the variable

MetaCLIP, MetaCLIP 2 and DataComp have byte-identical forward passes to a plain CLIP — step 1 through step 7 are the same diagram. Everything that separates them happened upstream, in what got crawled and what got kept. Step through them back to back.

Diet · how much data, and which data

Curation is a bigger lever than volume.

Two encoders trained on "2 billion image-text pairs" can differ by many points, because the pairs aren't the same pairs. The last three years of data work is mostly about filtering, balancing and de-duplicating — not about finding more of the internet.

Encoder	Data	Scale	The idea
CLIP 2021	WIT (private)	400M pairs	Scrape at scale, query-balanced by 500k text queries. ~32 epochs, so ≈12.8B samples seen.
OpenCLIP 2023	LAION-2B	2B pairs	Open reproduction; established clean scaling laws up to ~34B samples seen.
DataComp-1B	CommonPool	1.4B of 12.8B	Fix the training recipe, compete on the filter. A better 1.4B beats a worse 12.8B.
DFN-2B	filtered web	2B of ~5B	Train a small network whose only job is deciding what to keep. Filtering as a learned task.
MetaCLIP	CommonCrawl	400M / 2.5B	Reverse-engineer CLIP's curation: balance head and tail concepts by substring matching.
MetaCLIP 2 2025	worldwide web	300+ languages	Per-language metadata curation. Beats the English-only counterpart on English by 0.8 points — the "curse of multilinguality" is a data problem, not a capacity one.
SigLIP / SigLIP 2	WebLI	~10B images	SigLIP 2 trains multilingually across 100+ languages with explicit de-biasing, and adds staged self-supervised losses late in the run.
DINOv2	LVD-142M	142M images	No text at all. Retrieval-based curation from a raw pool, deduplicated against eval sets.
DINOv3 2025	curated web	7B params	An order of magnitude more curated images than v2, with Gram anchoring to keep dense features from decaying late in training.
InternViT-6B	multi-source	4.98B pairs	Progressive alignment: contrastive → generative → supervised, with an LLM in the loop.
Web-SSL 2025	MetaCLIP images captions discarded	up to 7B params	The controlled experiment: same images, no language. Language-free SSL matches CLIP on VLM tasks given enough scale and data diversity.
MiniMax-VL-01	image–caption	694M pairs	Proof that training a 303M encoder from scratch is still viable in 2025 if you have the data.

samples seen ≠ dataset size

Read the epochs

"400M pairs" and "12.8B samples seen" describe the same run. Compare compute budgets, not dataset cards, or you'll conclude the wrong thing about scaling.

resolution curriculum

Nearly everyone trains low, then finishes high

Cheap low-res epochs first, a short high-res phase at the end. UniViTAR makes this explicit as resolution curriculum learning; SigLIP 2 resumes at 95% completion to mint each resolution checkpoint.

contamination

Benchmarks leak into web crawls

Eval images appear in training pools. Serious releases deduplicate against benchmark sets; the difference between a deduplicated and non-deduplicated run is exactly the kind of gap that looks like an architectural win.

Field notes · the things that bite in practice

Fourteen nuances that don't fit in an architecture diagram.

artifacts

Your ViT has ~2% garbage tokens

Large ViTs — DINOv2, OpenCLIP, even supervised ones — develop high-norm outlier tokens in low-information background patches, roughly 10× the norm of their neighbours. The model repurposes redundant patches as scratch memory. Adding 4 dedicated register tokens gives it somewhere else to write, removes the artifacts and improves dense tasks. If your attention maps look speckled, this is why.

feature depth

Last layer vs penultimate vs all of them

The final block of a contrastive encoder is tuned for pooled similarity, so VLMs usually tap one layer earlier. Qwen3-VL goes further with DeepStack: inject multi-level ViT features into different LLM layers, keeping fine detail without lengthening the context.

training regime

Frozen, thawed, or trained from scratch

LLaVA freezes the encoder throughout — cheap, preserves pretrained features. Qwen unfreezes and trains at progressively higher resolutions. Unfreezing helps document and grounding tasks and risks catastrophic drift on everything else; it needs a much lower LR than the LLM.

geometry

Squashing changes what a chart says

Aspect distortion is not a cosmetic issue for charts, tables and UI screenshots — it changes slopes, column alignment and glyph shapes. NaFlex outperforms fixed-resolution SigLIP 2 mainly on OCR, document and screen benchmarks, and mostly at short sequence lengths where distortion is worst.

position

"Any resolution" has an envelope

Interpolated absolute embeddings degrade away from the training grid. NaFlex interpolates between its trained sequence lengths but extrapolates badly beyond them. RoPE-based schemes generalise further but still inherit the shapes they saw.

budget

Patch size or resolution — pick your spend

Both increase tokens. Smaller patches give finer sampling at the same field of view; higher resolution gives more field of view at the same sampling. For dense small text, resolution usually wins; for fine texture, patch size does. FlexiViT showed one model can support several patch sizes.

fusion

Complementarity is real but expensive

Cambrian-1 combined CLIP, SigLIP, ConvNeXt and DINOv2 and no single encoder matched the combination. Cost: roughly 4.3× a single encoder. SCOPE's mixture-of-encoder-experts routing recovers 24–49% of that. Distillation (RADIO) is the other answer: bake several teachers into one student and pay once.

connector

The projector matters less than you'd hope

MM1's ablations found connector design has modest impact next to encoder choice and data quality. NVLM held the encoder fixed and varied decoder-only vs cross-attention vs hybrid, and decoder-only matched the fancier options. A two-layer MLP is a fine default; spend your effort upstream.

context economics

Visual tokens compete with thinking tokens

A native-resolution 4K frame can exceed 20,000 tokens before compression. On a reasoning model that generates thousands of tokens per answer, image tokens crowd out the reasoning budget and inflate attention cost quadratically. Token efficiency is now as important as parameter efficiency.

grounding

Coordinate conventions differ between versions

Some models emit boxes normalised to a 0–1000 grid; Qwen2.5-VL moved to absolute pixel coordinates so outputs track real image scale. If your grounding suddenly lands in the wrong place after an upgrade, check the convention before the weights.

video

Frames are not free and time needs encoding

Most video VLMs group consecutive frames (Qwen2.5-VL uses 3D patches over frame pairs) and encode time explicitly. Qwen3-VL replaced learned temporal RoPE with literal textual timestamps — simpler, and better for "when did X happen".

evaluation

Compression benchmarks are misleading

VTC-Bench found that on standard VLM benchmarks, naive downsampling often matches sophisticated token-compression methods. Compression quality only separates on samples that genuinely need fine detail, dense text or spatial precision — which most benchmark samples do not.

encoder-free

The whole component may be optional

Fuyu-8B projects raw patches straight into the LLM. EVE/EVEv2 make it competitive with vision-centric supervision, and SAIL shows encoder-free scales comparably to modular VLMs. The catch is training compute: you're asking the LLM to learn perception from scratch.

safety

Bias enters through the encoder

Web-scale pretraining bakes in demographic, geographic and occupational skew — lower accuracy on darker skin tones, a North-America/Europe bias, uneven coverage of non-Latin scripts despite "multilingual" labels. As VLMs drive GUI agents and robots, an encoder failure becomes an action.

Three houses · the open recipes worth copying

Qwen, InternVL and Molmo solved the same problem three different ways.

All three care about documents, grounding and long video. Qwen builds its own native-resolution ViT and lets the token count float. InternVL keeps a fixed-resolution ViT and wraps it in tiling plus aggressive compression. Molmo does not build an encoder at all — it takes an off-the-shelf one, overlaps the crops so no patch is context-starved, and pools them with attention. Almost every other open VLM is a variation on one of these three.

Fig. 6 — three pipelines · click any stage

Qwen-VL series

Alibaba · native resolution

The through-line: stop resizing images, and make the encoder architecture look more like the language model.

Aug 2023 · Qwen-VL

A big borrowed encoder, resampled down hard

ViT-bigG at 1.9B parameters, initialised from OpenCLIP, running at a fixed 448px. A single-layer cross-attention resampler compresses the output to a fixed 256 tokens. The encoder is trained at progressively higher resolutions, then frozen for instruction tuning.

1.9B encoderfixed 448pxcross-attn resampler256 tokens/image

Sep 2024 · Qwen2-VL

Naive Dynamic Resolution — the pivot

The encoder shrinks to 675M and is rebuilt around NaViT-style native resolution: an image produces a variable number of tokens proportional to its area, with no resizing or cropping. A 2×2 merge in the connector cuts the sequence by 4×. M-RoPE splits rotary position into temporal, height and width components, so the same mechanism handles text, images and video.

675M encodernative resolutionM-RoPE (t,h,w)2×2 token merge

Feb 2025 · Qwen2.5-VL

The ViT gets rewritten in the language model's idiom

Window attention in most layers with a 112×112 pixel window (8×8 patches), and full self-attention at only four layers — so cost grows roughly linearly with patch count instead of quadratically. RMSNorm and SwiGLU replace LayerNorm and GELU. 2D-RoPE for spatial position, absolute-time alignment for video, and grounding output moved to absolute pixel coordinates.

window attn 112pxfull attn at layers 7,15,23,31RMSNorm + SwiGLU2D-RoPEdims snapped to ×28

Nov 2025 · Qwen3-VL

Fusion moves deeper into the LLM

The vision tower moves to a SigLIP 2 SO400M-class encoder, and the interesting change is where its features land. DeepStack injects multi-level ViT features into several LLM layers rather than only at the input — sharper fine detail and OCR without lengthening the context. Interleaved-MRoPE spreads temporal, height and width across the full frequency spectrum instead of partitioning it, and video timing becomes explicit text timestamps. Native 256K interleaved context.

SigLIP 2 SO400MDeepStack multi-level fusionInterleaved-MRoPEtext timestamps256K context

InternVL series

Shanghai AI Lab · tile & compress

The through-line: a big purpose-built encoder, then years of work on spending its tokens more cheaply.

Dec 2023 · InternVL 1.0

Close the scale gap by force

InternViT-6B: six billion vision parameters, built to match LLM scale rather than adapting a general-purpose encoder. Trained by progressive alignment across contrastive, generative and supervised stages on 4.98B image-text pairs, with a QLLaMA query-based connector. Fixed 448px input.

6B encoder4.98B pairsprogressive alignmentfixed 448px

Apr 2024 · InternVL 1.5

Dynamic high resolution

The tiling scheme that half the field copied: pick the tile grid whose aspect ratio is closest to the image, cut into 1–12 tiles of 448px, and prepend a thumbnail for global context. Documents suddenly work.

1–12 tilesclosest-aspect grid+ global thumbnail

Jul–Dec 2024 · InternVL 2.0 / 2.5

Pixel shuffle, and a small encoder that nearly keeps up

Each 448px tile is 1024 patches; a 2× space-to-depth shuffle compresses it to 256 tokens before the LLM sees anything. InternViT-300M ships alongside the 6B, and in practice the 20× parameter gap buys measurable gains on vision-centric tasks and very little on general benchmarks.

1024 → 256 tokens/tileInternViT-300M or 6Bup to 78B total

Apr 2025 · InternVL3

Native multimodal pretraining and cheaper visual positions

Instead of taking a finished text LLM and bolting vision on, language and multimodal ability are learned in a single joint pretraining stage — which also leaves text performance stronger than the base LLM it started from. V2PE gives visual tokens smaller position increments than text tokens, so long multimodal contexts don't blow through the position window.

joint pretraining stageV2PE variable incrementsMPO post-training

Aug 2025 · InternVL3.5

Route tiles by how much they're worth

The Flash variant adds a Visual Resolution Router: a per-tile decision between the standard 256-token compressor and a 64-token one, based on semantic richness, trained with a consistency objective so answers don't change with the routing. Roughly 50% fewer visual tokens with small losses on DocVQA/InfoVQA. Decoupled Vision-Language Deployment puts the encoder and the LLM on different GPUs to balance the load.

ViR router 256 ↔ 64~50% token cutconsistency-trainedDvD serving split

Molmo series

Ai2 · borrow the encoder, fix the crops

The through-line: the encoder is a commodity. What isn't commodity is how you crop, how you pool, and what data you collect yourself.

Sep 2024 · Molmo

Overlapping crops, because border patches are blind

Molmo tiles the image into square crops plus one downscaled global view — standard so far. The twist is that the crops overlap, so a patch at a tile border still gets encoded with its neighbours in view. The overlap region is then discarded before the connector, so the features passed on still tile the image exactly once. This directly attacks the failure mode that plain tiling creates: a brand name or a table rule split across a seam is now inside at least one crop that sees all of it.

CLIP ViT-L/14-336global view + tilesoverlapping cropsoverlap features dropped

Sep 2024 · Molmo connector

Two ViT layers, pooled by attention rather than stacked

Instead of taking one layer, Molmo concatenates patch features from the third-to-last and tenth-from-last ViT blocks — early enough to keep spatial detail, late enough to carry semantics. Each 2×2 patch window is then collapsed to a single token by a multi-head attention layer that uses the mean of the four patches as its query. Both choices were ablated and both beat the obvious alternatives: one layer, and naive concatenation.

layers −3 and −102×2 attention poolingmean patch as queryshared MLP projection

Sep–Dec 2024 · PixMo

The finding that the encoder barely matters

Molmo's ablations report that CLIP, SigLIP and MetaCLIP all perform roughly the same under the same recipe — which means a fully open stack (MetaCLIP encoder, OLMo language model, PixMo data) is achievable with no proprietary component anywhere. The contribution moved to the data: PixMo-Cap collected dense captions by having annotators describe images out loud for a minute or more and transcribing them, specifically to avoid distilling from a closed VLM. PixMo-Points added pointing supervision, so the model answers "where" by pointing at pixels instead of emitting a box.

encoder choice ≈ neutralspeech-collected captionspointing, not boxesfully open stack possible

Aug 2025 · MolmoAct

The same vision stack, pointed at robots

MolmoAct reuses the Molmo encoder and connector unchanged and adds spatial reasoning and action tokens — evidence that the crop-and-pool front end generalises past chat. The later MolmoAct2 swaps in the SigLIP 2 tower and keeps the connector design identical.

same ViT + connector3D spatial reasoningaction tokens

Jan 2026 · Molmo 2

SigLIP 2, a crop budget that changes at inference, and video

The tower moves to SigLIP 2 and the layer pair shifts to third-to-last and ninth-from-last. The crop budget is now explicitly a knob: K=8 overlapping crops during training, K=24 at inference — resolution you can buy after the model is trained. Video is sampled at 2 fps up to 128 frames (384 for long-context training), always including the final frame, and video frames pool over 3×3 windows instead of 2×2 to keep the token count down, sharing connector parameters with the image path. Visual tokens are interleaved with text timestamps for video and image indices for multi-image inputs. Backbones are Qwen 3 at 4B and 8B, or Olmo at 7B for a fully inspectable stack.

SigLIP 2 ViTlayers −3 and −9K=8 train → K=24 infer2 fps · 128 frames3×3 pooling for videoApache 2.0

The result

Data efficiency as the headline number

Molmo 2 at 8B beats the original Molmo at 72B on pointing and grounding, and was trained on 9.19M videos against the 72.5M used by Meta's PerceptionLM. Both numbers argue the same thing the series has argued since 2024: the encoder is not where the leverage is.

8B > the old 72B on pointing9.19M vs 72.5M videosopen weights + open data

What to take from the comparison. Native resolution wins on fidelity and loses on predictability — you cannot bound the token cost of an unknown image without clamping. Tiling wins on predictability and loses on geometry, since a tile grid is a quantised approximation of the aspect ratio and tile borders cut through content. Molmo splits the difference: keep the tiles, but overlap them so seams stop destroying content, and make the crop count a serving parameter rather than an architectural one.

The uncomfortable implication of Molmo's ablations is that two of these three teams spent enormous effort on the encoder to reach a place a third team got to with an off-the-shelf checkpoint and better cropping, pooling and data. If you are building a VLM rather than an encoder, that is probably the cheapest lesson on this page.

Wiring · the connector, and who uses what

The module between the encoder and the language model has fifteen shapes.

Visual embeddings and language embeddings live in different spaces, so something has to map one to the other. Formally the connector is a function ξ : V → T converting n visual features into m visual tokens. The whole design space is contained in the relationship between n and m: leave it alone, compress it losslessly, throw tokens away, or replace them with a fixed number of learned queries.

And yet: the connector matters less than almost anything else. MM1's ablations found connector design has modest impact next to encoder choice and training data quality. NVLM held the encoder fixed and compared decoder-only, cross-attention and hybrid designs — decoder-only matched the more complex options while being simpler to implement and scale. The field's answer over time has been to converge on a two-layer MLP and put the effort into compression instead, which is a different job wearing the same coat.

Encoder adoption, year by year

Computed from the table below. The shape of this chart is the single clearest summary of the last three years: a monoculture, a transition, and a new monoculture.

Which encoder is in which VLM

Ninety-plus models from the three surveys, with the vision encoder and connector each one actually ships. Search it, sort it, or filter by encoder family.

Date	Model	Org	Vision encoder	Connector

Compiled from the vision-encoder survey and the TIU review. Where two sources disagree — the TIU table lists Qwen2-VL with a CLIP ViT-G tower, the Qwen technical report describes a NaViT trained from scratch — the primary source wins. Dates are first public release.

Text-rich · where encoder decisions stop being academic

When the image is the text, every choice on this page gets amplified.

Documents, charts, tables, receipts, UI screenshots — the field calls this Text-rich Image Understanding, and it splits into two abilities: perception (detect, recognise, parse layout and formulas) and understanding (extract key information, answer questions). Before 2023 these were separate specialist models. Now they are one VQA-shaped pipeline, and the vision encoder is carrying weight it was never designed for.

Three architectures, one question: who reads the text?

This is the fork the general VLM literature skips entirely. You can let a vision encoder learn to read, you can hand it OCR output on a plate, or you can do both.

Architecture

The scoreboard says OCR-free won — but read the second table. Averaged over DocVQA, InfoVQA, ChartQA and TextVQA, the top five are all OCR-free: Qwen2-VL-72B (88.70), InternVL2.5-78B (87.73), InternVL2.5-38B (87.45), InternVL2.5-26B (85.85), DeepSeek-VL2-27B (85.40). On classic key-information extraction — FUNSD, CORD, SROIE — OCR-dependent models still lead, with scores like 96.5 on CORD and 97.8 on SROIE that no OCR-free model reports. Structured extraction from clean forms is a different problem from reasoning over a photographed page, and the winning architecture differs accordingly.

The compressor zoo

A document at usable resolution produces thousands of patches. Every serious document model puts something between the encoder and the LLM to cut that down — and the good ones exploit a property of documents specifically. Pick one to see what it does and what it assumes.

Tokens in: —
Tokens out: —
Ratio: —
Lossless?: —

Teaching an encoder to read

A CLIP encoder was never trained on a task where getting a digit wrong matters. Document models fix that in a modality-alignment stage built from OCR-derived tasks — the training data is generated from detection, recognition and parsing datasets, reformatted as VQA. These are the actual tasks.

The text-rich scoreboard

Average across DocVQA, InfoVQA, ChartQA and TextVQA. Worth reading for two things: how flat the top is, and where the small models land.

Mini-Monkey at 2B parameters scores 74.93 — within a point of MM1.5 at 7B. Scale is not what is producing these numbers; resolution handling and token compression are.

open problem

The token disproportion

A document page produces vastly more visual tokens than the question and the answer combined. Every efficiency technique in the zoo above exists because of that imbalance, and none of them has solved it.

open problem

Miniaturisation is working

Mini-Monkey reaches 74.93 on the text-rich average at 2B parameters — within a point of MM1.5 at 7B. The 7B default is a habit, not a requirement, and resolution handling plus token compression are doing the work that scale is being credited for.

open problem

Multilingual is barely started

Current models are tuned for English and a handful of high-resource languages. Non-Latin scripts and low-resource languages lag badly, and the benchmarks that would expose it — MTVQA, CC-OCR — are recent and thinly adopted.

Sequences · more than one frame

One image was the easy case.

Every trace and calculator above assumed a single picture. Real inputs are a photo album, a twelve-page PDF, a screen recording. Nothing about the encoder changes — each image still goes through the same forward pass — but three new problems appear at once: how the sequence is assembled, how position is assigned across images, and how you stay inside a context window when the answer is "multiply everything by N".

Assembling a multi-image sequence

Images are encoded independently and their tokens concatenated into one stream, wrapped in delimiter tokens so the language model knows where each one starts and stops. The interesting decision is what happens to the position ids.

Fig. 7 — the assembled sequence · drag the image count, switch the position scheme

images in the prompt 3

position ids

What can a text token actually see?

Two architectures answer this differently, and the difference decides whether a model can compare image 1 with image 3 at all.

Fig. 8 — attention scope across images · click any token

architecture

Multi-page documents are the hard version of this. Most document models are single-page. The multi-page ones typically retrieve one relevant page and process it alone, which loses every dependency that crosses a page boundary. The workarounds are architectural: DocOwl 2's H-Reducer compresses page-level features while keeping cross-page reasoning, PDF-WuKong sparse-samples only the relevant text and diagrams rather than reading everything. Long-document understanding remains one of the field's openly acknowledged gaps.

Video: sampling is the first lossy step, and nobody reports it

A video is not encoded. A handful of frames sampled from it are. Everything between those frames is discarded before the model exists in the loop — and the sampling rate and frame cap are serving parameters, not model properties.

Fig. 9 — video token budget · drag anything

preset

clip length 60 s

sample rate 2 fps

tokens / frame 391

temporal packing

Frames available: —
Frames encoded: —
Seconds discarded: —
Visual tokens: —
Of a 128K window: —

Frame budgets across the field

The spread is three orders of magnitude, and it tracks how the model handles position and compression far more than how big its encoder is.

Model	Max frames	Per-frame strategy	Temporal position
Video-LLaVA	8	LanguageBind joint image-video space	implicit, in the ordering
VideoLLaMA 2	16	spatial-temporal aggregation	implicit
LLaVA-NeXT-Video	32	AnyRes tiling + temporal	implicit
LLaVA-OneVision	32	one unified path for images and frames	implicit
InternVL 2.0	64	frame sampling + pixel shuffle	implicit; V2PE from InternVL3
VideoLLaMA 3	64	temporal pooling · SigLIP/DFN	implicit
Oryx	64	dynamic resolution per frame	implicit
ELVA	64	encoder-free, direct video	implicit
Video-ChatGPT	100	spatial-temporal pooling	implicit
Molmo 2	128 · 384	2 fps · 3×3 frame pooling, shared connector	explicit text timestamps
Qwen2-VL	768	NaViT native resolution per frame	M-RoPE temporal component
Qwen2.5-VL	768	two frames per 3D patch	M-RoPE aligned to absolute time
Qwen3-VL	256K ctx	SigLIP 2 + DeepStack	explicit text timestamps, replacing T-RoPE
LongVA	2000	long-context extrapolation	implicit
MiniCPM-V 4.5	—	3D-Resampler	implicit

sampling

The last frame is special

Molmo 2 always includes the final frame regardless of the sampling schedule, because most video players leave it on screen after playback ends — so a user asking "what happened" is often looking at exactly that frame. A one-line heuristic that comes from thinking about the user rather than the data.

temporal position

From learned time to literal timestamps

Qwen2-VL gave rotary position a temporal axis. Qwen2.5-VL aligned it to absolute time, so a model can answer "at what second". Qwen3-VL then replaced the mechanism with text timestamps interleaved into the token stream — the model reads the time the way it reads everything else.

cost

Video is where the quadratic bites

Token count grows linearly with frames, but attention inside the LLM grows with the square of total context. Doubling the frame count roughly quadruples the attention cost — which is why frame caps exist and why every video model compresses harder per frame than its image path does.

evaluation

Video-MME is encoder-sensitive

Unlike general VQA, video analysis sits in the high-sensitivity band: encoder choice and per-frame resolution directly determine the ceiling. Two models with identical language backbones can differ enormously here purely on how many pixels per frame survive.

reporting

Nobody tells you the sampling rate

A video benchmark score is meaningless without the fps and frame cap used to produce it. The same checkpoint evaluated at 64 frames and at 8 frames is effectively two different systems, and the difference is larger than most architectural changes on this page.

multi-image

Grounding needs to name the image

With several images in context, "the cat on the left" is ambiguous until the model can refer to image indices. Molmo 2 interleaves explicit image indices for multi-image inputs and trains on curated image sets with pointing supervision, so references resolve to a specific picture.

Calibration · how to tell encoders apart

Most benchmarks can't hear the difference.

There is no standard evaluation protocol for vision encoders. Studies vary the LLM, the connector, the resolution, the training data and whether anything was fine-tuned — then report a single number. Before believing an encoder comparison, check that everything except the encoder was actually held constant.

Encoder sensitivity: where swapping the encoder changes the answer

DocVQA · OCRBench · ChartQAtext & document understanding

high

RefCOCO · TallyQA · RealWorldQAgrounding, counting, spatial

high

BLINKfine-grained visual perception

high

Video-MMEvideo analysis

high

ADE20K · COCOstandalone dense prediction

high

MMBench · MME · MMMUmulti-ability, vision + language

medium

POPE · MMStar · SEED-Benchhallucination, mixed reasoning

medium

VQAv2 · GQAgeneral VQA — saturated

low

ScienceQAknowledge-dominated

low

Bar length is the degree to which benchmark score tracks encoder choice rather than LLM capability, connector or training data. Swapping CLIP ViT-L/14 for SigLIP 2 might move VQAv2 by a point or two and move OCRBench by tens of points.

The cost side of the ledger

Encoder	Params	GFLOPs	VRAM	ImageNet 0-shot	VQAv2	Read as
CLIP ViT-B/16	86M	17.6	0.4 GB	68.3	76.8	the floor
CLIP ViT-L/14	304M	81.1	1.2 GB	75.5	79.2	the 2023 default
SigLIP-B/16	93M	17.9	0.4 GB	78.4	78.1	beats CLIP-L at ⅓ the params
SigLIP-SO400M	400M	95.8	1.5 GB	83.2	81.7	the safe modern pick
SigLIP 2 SO400M	400M	98.3	1.6 GB	84.1	82.4	the 2025 default
InternViT-300M	304M	82.6	1.3 GB	79.8	80.3	the distilled sibling
InternViT-6B	5.9B	1,547	24 GB	88.2	82.1	19× the FLOPs, −0.3 on VQAv2
DINOv2-L	304M	81.6	1.2 GB	86.3	73.2*	*needs text alignment first
Cambrian (4 encoders)	2.1B	412	5.8 GB	—	77.8	≈4.3× cost, pays off on spatial

Standalone evaluation

ImageNet zero-shot — semantics and text alignment; expect >80 for anything modern
ADE20K mIoU, COCO mAP — dense prediction, where SSL encoders shine
Linear probing / kNN — feature quality without a decoder confusing things
DocBank, DocLayNet, RICO-SCA — document and UI structure
Caveat: good classification features don't automatically transfer to language-conditioned tasks

VLM-integrated evaluation

Real utility, but confounded by LLM, connector, data and resolution
Fix everything but the encoder, or the comparison is meaningless
Report realised token count per image alongside accuracy
Include at least one high-sensitivity benchmark or you'll measure nothing
Probe MMVP-style pairs to catch inherited blind spots

Selection · what to actually use

Start with SigLIP 2. Deviate only for a reason you can name.

General-purpose VLM

SigLIP 2 SO400M

Best training methodology per parameter, multilingual, decent dense features. 400M is enough for almost everything that isn't document-shaped.

Documents, receipts, charts, UI

Native or NaFlex, plus tiles

Aspect preservation and real resolution beat encoder size here. A NaViT-style encoder or tiling + pixel shuffle; consider a document-specialist encoder like DeepEncoder or DAVE.

Counting, depth, referring, robotics

Add DINOv2 or v3

Self-supervised features carry geometry that contrastive training discards. Fuse with SigLIP rather than replacing it — you need both semantics and structure.

You want fusion without paying 4×

A distilled agglomerate

RADIO-family models bake CLIP, DINO and SAM teachers into one backbone with one forward pass, and handle arbitrary resolutions and aspect ratios.

Tight latency or edge deployment

SigLIP 2 Base, or FastVLM-style hybrids

86M–303M keeps most of the quality. Hybrid CNN-ViT designs cut token count directly, which matters more than encoder FLOPs once the LLM dominates.

You're tempted by a 6B encoder

Probably don't

Twenty times the parameters buys measurable gains on vision-centric tasks and roughly nothing on general benchmarks. Spend the budget on resolution, data quality or a second complementary encoder.

A serviceable default stack, end to end: SigLIP 2 SO400M with native-aspect intake, a token budget you set deliberately per surface (documents high, chat low), a two-layer MLP connector, features tapped from the penultimate layer, and the encoder frozen until you have a measured reason to thaw it. Then measure on one high-sensitivity benchmark before and after every change.

Where this is heading

Four unresolved tensions define the next couple of years. Whether encoder-free models close the training-compute gap and make the whole component optional. Whether unified visual-text tokenisation (Chameleon, Emu3) can capture spatial structure as well as continuous features do. What happens past 4K, where a single frame is 50,000+ tokens before compression. And whether the field converges on universal encoders or fragments into document, UI, video and robotics specialists.

The stable part, whatever wins: training objective innovation, resolution flexibility, and complementarity between differently-supervised features. Those three have explained every real gain since 2021, and none of them is a parameter count.

References · the papers behind every claim above

References

The intake bench

Pixels in, a sequence of vectors out. Everything else is a choice.

The six knobs that define an encoder

Nine things happen to your image before a single attention head sees it.

Decode & orient

Resize — and the algorithm matters more than you'd think

Aspect handling: squash, crop, or pad

Snap to a patch multiple

Pad to a packed sequence — and build the mask

Normalise with the right constants

Patchify & project

Add position information

Set the token budget

Five ways to feed a 4-megapixel document into a 224-pixel brain.

Resize to 224 or 336 and accept the loss

Bicubic-interpolate the position grid, then fine-tune

Cut the image into tiles, encode each, add a thumbnail

InternVL: dynamic tiles, then pixel shuffle

NaViT / Qwen: don't resize at all

Spend tokens where the information is

Token cost calculator

What tiling actually does to an image

Pixel shuffle: the compression trick hiding in every InternVL config

Training methodology beats parameter count, and it isn't close.

Contrastive: softmax coupling vs sigmoid independence

Self-supervised: learning geometry with no words at all

The objective decides the failure mode

What contrastive encoders see

What self-supervised encoders see

There are about a dozen families. Everything else is a variant.

What actually happens between a JPEG and a token.

Divergence happens early

Special tokens are not cosmetic

The architecture is rarely the variable

Curation is a bigger lever than volume.

Read the epochs

Nearly everyone trains low, then finishes high

Benchmarks leak into web crawls

Fourteen nuances that don't fit in an architecture diagram.

Your ViT has ~2% garbage tokens

Last layer vs penultimate vs all of them

Frozen, thawed, or trained from scratch

Squashing changes what a chart says

"Any resolution" has an envelope

Patch size or resolution — pick your spend

Complementarity is real but expensive

The projector matters less than you'd hope

Visual tokens compete with thinking tokens

Coordinate conventions differ between versions

Frames are not free and time needs encoding

Compression benchmarks are misleading

The whole component may be optional

Bias enters through the encoder

Qwen, InternVL and Molmo solved the same problem three different ways.

Qwen-VL series

A big borrowed encoder, resampled down hard

Naive Dynamic Resolution — the pivot

The ViT gets rewritten in the language model's idiom

Fusion moves deeper into the LLM

InternVL series

Close the scale gap by force

Dynamic high resolution

Pixel shuffle, and a small encoder that nearly keeps up

Native multimodal pretraining and cheaper visual positions

Route tiles by how much they're worth

Molmo series

Overlapping crops, because border patches are blind

Two ViT layers, pooled by attention rather than stacked

The finding that the encoder barely matters

The same vision stack, pointed at robots

SigLIP 2, a crop budget that changes at inference, and video

Data efficiency as the headline number

The module between the encoder and the language model has fifteen shapes.

Encoder adoption, year by year

Which encoder is in which VLM

When the image is the text, every choice on this page gets amplified.

Three architectures, one question: who reads the text?

The compressor zoo

Teaching an encoder to read

The text-rich scoreboard