How Models Know Where They Are

01The symmetry to break

Write attention out and the problem is immediate. For a set of token vectors x₁ … x_n, attention computes softmax(QKᵀ/√d)V. Permute the rows of the input and every one of those matrices permutes with it. The output is the same set of vectors, in the same permuted order.

Formally, attention is permutation equivariant: f(Px) = P·f(x) for any permutation matrix P. That is a lovely property for point clouds and a fatal one for language. The MLP, the residual stream, LayerNorm — none of them look across tokens, so none of them fix it. The only two places order can enter a transformer are (a) something you add to or apply to the token representations, and (b) the attention mask itself. Almost every method in this guide is a variation on (a). One important family, which we get to in §07, quietly exploits (b).

It's worth being precise about what we want from a positional scheme, because the schemes differ almost entirely in which of these they prioritise:

• Injectivity — different positions should be distinguishable.
• Translation equivariance — the interaction between tokens at i and j ideally depends on i − j, not on where the pair sits in the document.
• Locality bias — nearby tokens should be easy to attend to; a useful prior, but a liability if you need retrieval across 500k tokens.
• Length generalisation — behaviour at positions never seen in training should degrade gracefully.
• Cheapness — no extra O(n²) tensors, KV-cache-friendly, and compatible with FlashAttention.

No scheme wins on all five. RoPE, which is what almost everyone ships, trades length generalisation for the other four, and then we spend enormous effort buying the fifth back.

02Absolute: a table of addresses

The simplest fix, introduced for convolutional seq2seq by Gehring et al. (2017) and adopted by BERT, GPT-2, ViT, and CLIP: keep a learned matrix P ∈ ℝ^L×d, look up row p for the token at position p, and add it to the token embedding before layer 1.

It works, it is trivial to implement, and it has two structural problems. First, P has a hard maximum length — position 513 in BERT is not a row that exists. Second, nothing in the parameterisation ties row p to row p+1; adjacency has to be learned from data, position by position, and positions that are rare in training are poorly estimated. That's the origin of the long tail of weirdness people find when probing GPT-2's position embeddings near its context limit.

A subtler issue, pointed out by Ke et al. (2021, TUPE): adding position to the token embedding means the QKᵀ expansion contains four terms — content-content, content-position, position-content, position-position — and the two cross terms are largely noise. TUPE untangles them and computes the positional term with its own projection. DeBERTa (He et al., 2021) makes the same observation and keeps content and relative position disentangled throughout, folding absolute position back in only in the last few layers for tasks that need it.

Absolute learned embeddings are not dead. Every CLIP and SigLIP vision tower still ships one, and Gemma 3's SigLIP encoder does too. In vision the fixed-length problem shows up as a fixed resolution, which we deal with in §08.

03Sinusoidal: a frequency code

The original transformer used no parameters at all. Vaswani et al. (2017) defined, for dimension pair i,

Each pair of dimensions is a clock. Pair 0 ticks with wavelength 2π; the last pair has wavelength 2π·10000. Reading the whole vector is like reading a smooth, continuous odometer: the fast dials disambiguate nearby positions, the slow dials carry coarse "where in the document" information. This is exactly the binary counter analogy Vaswani's paper gestures at, made differentiable.

Two properties matter. The dot product PE(p)·PE(p+k) depends only on k and decays (non-monotonically) with it — so a model can read off relative offset from a single inner product. And PE(p+k) = R_k PE(p) for a fixed block-diagonal rotation R_k, meaning a linear layer can in principle implement "shift by k". Hold onto that second fact: RoPE is what you get when you stop adding that rotation and start applying it.

Sinusoidal encodings are defined for every integer, so they extrapolate in the trivial sense — but the model's learned use of them doesn't, which is why length generalisation with sinusoidal PE is poor in practice. They survive today mostly in encoders where length is fixed: Whisper's audio encoder, DETR's 2D variant, MAE's fixed 2D sin-cos grid.

04Relative: distances, not addresses

If what matters is i − j, encode i − j directly. Shaw et al. (2018) did it first: add a learned vector a_ij, indexed by the clipped relative distance, to the keys (and optionally the values) inside the attention computation. Clipping at ±k keeps the table small and gives free extrapolation — everything beyond k shares one bucket.

Transformer-XL (Dai et al., 2019) reworked this into a sinusoidal relative form with separate learned global content and position biases, which is what made segment-level recurrence coherent across segment boundaries. T5 (Raffel et al., 2020) stripped it to the bone: a single learned scalar per (head, bucket), added to the attention logit, with 32 log-spaced buckets so that distance 8 and distance 9 get their own bias while distance 400 and 450 share one. The bias table is shared across layers. It is astonishingly cheap and still one of the best length-extrapolating schemes anyone has measured.

ALiBi (Press et al., 2022) removes even the learning: bias the logit by −m_h·|i − j|, where the slope m_h is a fixed geometric sequence over heads. Some heads get a steep slope and see only a local window; some get a shallow slope and see far. Train at 1k, run at 10k, and perplexity holds — the property that made BLOOM and MPT choose it.

The family has kept growing. KERPLE (Chi et al., 2022) generalises ALiBi to a family of conditionally positive-definite kernels with learned parameters (power and logarithmic variants); Sandwich (Chi et al., 2023) derives a similar decay from the cross term of sinusoidal encodings. FIRE (Li et al., 2024) replaces the bias table with a small MLP applied to a progressively normalised distance — (i−j)/max(i, L) — so the same function covers any length, which gives the best length generalisation in that paper's benchmark suite.

05RoPE: rotate, don't add

RoFormer (Su et al., 2021) asks for a function f(x, p) such that ⟨f(q, m), f(k, n)⟩ depends on q, k, and m − n only. In two dimensions the answer is forced: rotate by an angle proportional to position. Rotations compose by adding angles, and an inner product of two rotated vectors depends only on the angle difference.

Generalise by splitting the head dimension into d/2 planes and rotating plane i by m·θ_i, with θ_i = base^−2i/d — the same frequency ladder as sinusoidal encodings, now used as rotation speeds rather than as an added vector.

Three consequences worth internalising:

It is norm-preserving. Unlike additive schemes, RoPE cannot change the magnitude of q or k, only their phase. There is no "position budget" competing with content in the residual stream.

The frequency bands do different jobs. High-frequency planes complete a full rotation every few tokens, so they encode fine local offset; low-frequency planes barely rotate across the whole context, so they pass content through nearly untouched. Barbero et al. (2024) show empirically that heads use the two ends for different purposes — low frequencies carry semantic channels that RoPE is effectively not rotating, and they propose p-RoPE, truncating the lowest frequencies entirely. That is the same instinct behind partial rotary in GPT-NeoX and Phi, which apply RoPE to only 25–40% of each head's dimensions.

Decay is a soft prior, not a guarantee. The often-quoted "long-term decay" bound comes from Abel summation over the frequency ladder; it bounds an upper envelope, and real attention heads routinely punch through it to retrieve distant tokens. Treat it as an inductive bias, not a law.

Implementation detail that bites people: there are two conventions for which dimensions pair up. The paper interleaves (x₀,x₁), (x₂,x₃), …; most HF implementations split in half and pair (x_i, x_i+d/2) via the rotate_half trick. They are related by a permutation of the head dimension and are not interchangeable across checkpoints. Similarly, DeepSeek's MLA needs a decoupled RoPE head — a few dimensions carried outside the low-rank compressed KV — because you cannot rotate a latent that will be re-expanded by a position-independent matrix.

06Stretching RoPE past its training length

RoPE is defined for any m, but a model trained to 4k has never seen the rotation angles that positions beyond 4k produce, and it falls off a cliff. The fix is to change the mapping from position to angle at inference time — and this small design space has produced a surprising amount of literature.

Position Interpolation (Chen et al., 2023): divide every position by the extension factor s, so position 8000 in a 2k→8k extension is presented as 2000. All angles stay in-distribution; the cost is that adjacent tokens are now crowded together in phase, hurting fine local discrimination. A few hundred steps of fine-tuning fixes most of it.

NTK-aware scaling (a community result from bloc97, later formalised) observes that PI is wasteful: high-frequency planes were never the problem — they wrap around constantly anyway. Instead of scaling positions, scale the base, which stretches low frequencies a lot and high frequencies barely at all. This is why "just bump rope_theta" works so well and why Llama 3 ships with θ = 500{,}000 instead of 10,000.

YaRN (Peng et al., 2023) makes the split explicit. Compute each plane's wavelength; if it is much shorter than the trained context, extrapolate it untouched; if it is longer, interpolate it; ramp smoothly in between. YaRN also adds a temperature on the attention logits (1/√t) to compensate for the entropy shift, which is the part people forget to port.

LongRoPE (Ding et al., 2024) stops hand-designing the ramp and searches for a per-dimension rescaling with evolutionary search, plus special handling of the first few tokens, reaching 2M context. PoSE (Zhu et al., 2024) trains on skipped position ids so a short-context training run covers long-context angles. SelfExtend (Jin et al., 2024) does it training-free with grouped (floor-divided) positions for distant tokens and exact positions inside a neighbour window.

On the theory side, Liu et al. (2024) and Men et al. (2024) both argue that base θ is not free: too small and you cannot extrapolate; too large and the lowest-frequency planes never complete a rotation within the training window, so the model never learns to use them, and long-range retrieval silently degrades. There is a lower bound on θ implied by your target context length. That's the real reason the industry drifted 10k → 500k → 1M as context targets grew, rather than fashion.

07NoPE, and the hybrids that won

Now the twist. A decoder-only transformer with no positional encoding at all is not permutation equivariant, because the causal mask isn't. Token i attends over exactly i+1 tokens, and that count is recoverable: a head that attends roughly uniformly produces an average whose magnitude scales like 1/(i+1). Haviv et al. (2022) showed such models learn position anyway and can be probed for it; Kazemnejad et al. (2023) showed that on length-generalisation benchmarks NoPE beats ALiBi, T5-bias, and RoPE, and proved it can represent both absolute and relative schemes.

NoPE alone doesn't scale to production long-context models — the implicit signal is weak and diffuse. But the hybrid does. Llama 4's "iRoPE" interleaves RoPE-free global attention layers with local RoPE layers, with inference-time attention temperature scaling, targeting very long context. Cohere's Command A uses the same shape under the name RNoPE-SWA: sliding-window layers with RoPE for local structure, full-attention layers with no positional encoding for retrieval. Gemma 2 and 3 keep RoPE everywhere but split the difference differently — 5 local sliding-window layers per global layer, with θ = 10{,}000 locally and θ = 1{,}000{,}000 globally.

The pattern across all three: local layers need positional precision; global retrieval layers are hurt by a distance prior. That framing also explains attention sinks. Xiao et al. (2024, StreamingLLM) found that keeping the first few tokens' KV entries is necessary for streaming — and that when you evict, you must assign positions within the cache, not from the original text. Barbero et al. (2025) argue the sink is a learned no-op that prevents over-mixing. Position and sinks are the same story told twice.

08Two dimensions: position in vision encoders

ViT flattens a H/p × W/p patch grid into a raster sequence and adds a learned 1D embedding table — one row per grid cell, no 2D structure imposed. Dosovitskiy et al. ablated 2D-aware variants and found no gain, which everyone quotes; the more interesting finding is what the table learns. Plot the cosine similarity between one cell's embedding and every other cell's and you get a bright blob at that cell, fading outward, with faint bands along the same row and column. The network reconstructs 2D locality from scratch.

The fixed-table problem returns as a fixed input resolution. Fine-tuning at higher resolution requires interpolating the position grid — reshape to 2D, bicubic upsample, flatten. It works (everyone does it, from ViT to DINOv2 to SigLIP2-NaFlex), but it is a hack, and the details matter: DINOv2's implementation notoriously needed antialiasing to reproduce reported numbers, and interpolating with an odd class token in the sequence is a classic off-by-one bug.

The alternatives, roughly in order of adoption:

Fixed 2D sin-cos. Split the channels in half, apply the 1D sinusoidal formula to y in one half and x in the other, concatenate. Used by MAE, and by essentially every DiT-style model. No parameters, extends to any grid size, no learned locality.

Relative position bias. Swin (Liu et al., 2021) keeps a learned bias table indexed by relative (Δy, Δx) within each window. Swin-V2 replaces the table with a small MLP over log-spaced coordinates so that a model trained with 8×8 windows transfers to 24×24 windows — continuous relative position bias.

Conditional PE. CPVT (Chu et al., 2021) generates position from the features themselves with a depthwise convolution inserted after the first block. Any resolution, translation equivariant, free.

2D RoPE. The now-dominant answer. Axial RoPE gives half the head dimensions the rotation y·θ_i and half the rotation x·θ_i. It is exact, resolution-free, and adds nothing to the attention kernel. Heo et al. (2024) studied it carefully for ViTs and additionally proposed RoPE-Mixed, where each head learns its own complex frequency per axis, so basis functions need not be axis-aligned — diagonal and oriented patterns become representable, which helps detection and segmentation more than classification.

Adoption tells the story: EVA-02 was early with 2D RoPE; Pixtral's encoder uses it; Qwen2-VL and Qwen2.5-VL use 2D RoPE inside the ViT (with windowed attention in 2.5); SAM 2's memory attention uses it; and DINOv3 (2025) dropped DINOv2's learned table entirely for axial RoPE with random coordinate jittering during pretraining, which is what lets its features stay stable up to 4k-resolution inputs.

09Fourier features, NeRF, and SAM

There is a parallel lineage that never went through the transformer at all. Tancik et al. (2020) asked why a coordinate MLP fits blurry images and answered it with NTK theory: a plain ReLU MLP on raw (x, y) has a kernel that is far too wide, so it cannot represent high-frequency detail. Map the coordinate through random Fourier features first,

and the induced kernel becomes a stationary Gaussian whose width you control with σ. Too small and you get the same blur; too large and the fit is noise. This is the positional encoding in NeRF (which uses a deterministic log-linear frequency ladder rather than a random B), in Perceiver, and — importantly for anyone doing segmentation — in SAM.

SAM's design is the clean illustration of why this family exists. The image encoder produces a 64×64 embedding grid; the prompt encoder has to turn an arbitrary continuous click at, say, (0.413, 0.772) into a vector the mask decoder can compare against those grid cells. Both use the same random Fourier encoding with the same B, so a dot product between a prompt embedding and an image-grid embedding is a smooth similarity kernel peaked at the clicked location. A lookup table cannot do this — clicks are not grid-aligned. Li et al. (2021) generalise the idea further with learnable Fourier features for arbitrary multi-dimensional position.

DETR's 2D sinusoidal encoding is the same idea with a fixed rather than random basis, and it is worth noting that it is added at every attention layer, not once at input — a small detail that matters when object queries need to keep re-reading spatial position.

10M-RoPE: three axes at once

Now put an image into a language model. The LLM's RoPE is 1D, so the naive approach — flatten the patch grid into the token stream and let it inherit sequential position ids — tells the model that the patch at the end of row 1 and the patch at the start of row 2 are adjacent, when they are vertically distant. It also makes an image consume position ids proportional to its token count, so a 4-megapixel image can eat a large fraction of the context window's positional range before any text is read.

Qwen2-VL (Wang et al., 2024) introduced M-RoPE to fix both. It decomposes the position embedding into three components — temporal, height, and width. For text, all three use identical position ids, which makes M-RoPE functionally identical to 1D RoPE. For images the temporal id is constant across visual tokens, while height and width ids come from each token's spatial location. For video, the temporal id increments per frame while height and width follow the image scheme. Mechanically: the head's frequency ladder is partitioned into three contiguous sections (mrope_section = [16, 24, 24] over the 64 half-dimensions of a 128-dim head), and each section is driven by a different axis.

Qwen2.5-VL then fixed the temporal axis. In Qwen2-VL the temporal ids were tied to the frame index, which ignored how fast content changed and the absolute timing of events; Qwen2.5-VL aligns the temporal component with absolute time so that the intervals between temporal ids are consistent across videos sampled at different FPS. Two clips of the same scene at 1 fps and 4 fps now produce the same temporal coordinates.

Qwen3-VL changed the frequency allocation. Partitioning the dimensions into contiguous temporal, horizontal, and vertical subspaces gives each axis a different slice of the frequency spectrum — an imbalance that later work showed degrades long-video understanding. Qwen3-VL interleaves the t/h/w components across the embedding dimensions so each axis is represented uniformly across both low- and high-frequency bands. Under the old layout, t got only the highest-frequency planes, so temporal position wrapped around constantly and had no slow dial to carry long-horizon structure. A practical side effect: because interleaved M-RoPE's position ids grow more slowly than vanilla RoPE, a smaller YaRN factor is needed — going from 256K to 1M wants factor 2 or 3, not 4. Qwen3-VL also moves from T-RoPE toward explicit textual timestamp tokens for video, on the grounds that implicit temporal encoding alone was not enough for precise absolute grounding.

VideoRoPE (2025) arrives at a compatible conclusion from a different direction, arguing for low-frequency temporal allocation, a diagonal spatiotemporal layout that keeps text–video relative distances sensible, and adjustable temporal spacing.

11Position bookkeeping in VLM pipelines

Even in models that don't touch the LLM's RoPE, the way images are cut up and serialised is itself a positional design decision, and it varies wildly.

Fixed grid. LLaVA-1.5 resizes to 336×336, gets 24×24 patches, and the CLIP tower's position table matches exactly. Simple, and it throws away resolution.

AnyRes / dynamic tiling. LLaVA-NeXT and InternVL cut a high-resolution image into tiles that each go through the encoder at its native resolution, plus a downscaled thumbnail for global context. Each tile restarts its position embedding at (0,0), so the encoder has no idea how tiles relate; the LLM has to infer layout from the token order and from inserted newline/separator tokens. That's a real weakness, and it's why the number and arrangement of tiles is such a sensitive hyperparameter.

Patch-n-pack. NaViT (Dehghani et al., 2023) keeps native aspect ratio, packs variable-length sequences into one batch, and uses factorised (separate x and y) embeddings — with a fractional variant that normalises coordinates to [0,1] so any resolution maps into the same table. SigLIP2's NaFlex variant and Idefics2/3 follow this lineage.

Explicit structure tokens. Pixtral uses 2D RoPE in the encoder and inserts [IMG_BREAK] at the end of every patch row so the LLM's 1D stream carries the line structure. Fuyu goes furthest: no vision encoder at all, patches projected straight into the LLM, with newline tokens marking row ends. Molmo's approach is similar in spirit — overlapping crops plus explicit row/column marker tokens.

Compressing the positional footprint. InternVL3's V2PE gives visual tokens a smaller position increment than 1 (e.g. 1/16 per token), on the argument that visual tokens are individually less information-dense than text tokens and shouldn't consume the same positional budget. It buys a lot of effective context.

Nothing special at all. Chameleon and Emu3 tokenise images into discrete codes and run one ordinary 1D RoPE stream over everything. It works, and it makes the point that M-RoPE is a prior, not a requirement.

12The atlas: who uses what

Filter by family or search for a model, a scheme, or a keyword like ALiBi, 2D RoPE, or tiling.

13Nuances, failure modes, choices

Position is not the same as order

CoPE (Golovneva et al., 2024) makes the sharpest version of this point: every scheme above counts tokens, but the useful unit is often a sentence, a paragraph, or the i-th occurrence of something. CoPE computes a gate per key and takes a cumulative sum, so "position" becomes a learned, context-dependent, fractional quantity — and it solves counting and selective-copy tasks that token-counting schemes fail. DAPE makes the bias itself a function of the attention values. Randomized positional encodings (Ruoss et al., 2023) attack length generalisation from the data side by sampling a sorted random subset of a much larger position range during training.

Positional bias is a real bug, not a curiosity

Liu et al. (2023) — "Lost in the Middle" — showed U-shaped retrieval accuracy over document position. Some of that is the data distribution, some is the decay prior baked into RoPE and ALiBi. If your RAG pipeline's recall depends on where in the prompt the evidence landed, this is why, and reranking to put evidence at the edges is a legitimate (if inelegant) mitigation.

Packing and masking must agree with position ids

If you pack multiple documents into one sequence, position ids must restart per document and the mask must block cross-document attention. Forgetting the reset is one of the most common silent training bugs — the loss looks fine, the model just learns a slightly wrong notion of distance. The same applies to multimodal packing, where M-RoPE ids are three-dimensional and cannot be derived from a cumulative token count.

Things that get quietly broken in inference stacks

• Cache eviction: with RoPE the cached keys are already rotated. If you drop tokens, the remaining keys keep their original angles — which is right for relative semantics but wrong if your kernel assumes contiguous ids. StreamingLLM's fix is to assign positions by cache slot.
• Speculative decoding and prefix reuse: both assume position ids you may have redefined.
• Float precision: compute the RoPE cos/sin table in fp32. Doing it in bf16 at position 500,000 loses enough angular resolution to visibly hurt long-context retrieval.
• Interleaved vs. half-split RoPE: mismatched conventions between a converter and a kernel produce a model that is coherent for ~20 tokens and then drifts. It's a distinctive failure signature.

What I would actually pick

• Text decoder, ≤128k: RoPE, half-split, θ chosen for your target length (500k–1M), fp32 tables, fine-tune at the target length rather than relying on inference-time scaling.
• Text decoder, very long context: local/global interleaving — sliding-window RoPE layers plus a minority of full-attention layers with either no PE or a very large θ.
• Vision encoder: axial 2D RoPE with coordinate jittering. It's resolution-free, it costs nothing, and it removes the interpolation hack permanently.
• Continuous coordinates / prompts / fields: Fourier features, tuned σ.
• VLM: M-RoPE with interleaved frequency allocation, temporal ids tied to absolute seconds, and — if long documents matter — a sub-unit position increment for visual tokens.

14References

Grouped by theme rather than chronology. Every entry links to the paper.