The Mechanics of LLM Inference

The dependency that changes everything

A trained transformer is a function. You hand it a sequence of tokens and it hands back, for every position, a probability distribution over what comes next. During training you run that function on batches of known text, compare its predictions against the tokens that actually followed, and push the weights around. During inference you run the same function, with the same weights, doing the same arithmetic in the same kernels.

And you get a completely different machine. Not a slower one, or a smaller one — a machine with different bottlenecks, different memory behaviour, different scaling laws and a different economics. Nearly all of that difference traces back to one structural fact, which is worth stating as plainly as possible.

The autoregressive factorisation p(x₁, …, x_T) = ∏_t p(x_t | x_<t)

To sample x_t+1 you must condition on x_t. To have x_t you must have already sampled it. The token you are about to generate is a function of the token you just generated, so the forward passes cannot be run concurrently — not with more GPUs, not with a better kernel, not with a cleverer scheduler. Generating 500 tokens means 500 dependent forward passes through the entire model, in order.

During training this dependency is invisible. The target sequence is already known, so you feed the whole thing in at once and let the causal mask handle the conditioning; every position is predicted in parallel in a single pass. This is teacher forcing, and it is doing more work for you than most people realise. It converts an inherently serial process into an embarrassingly parallel one, which is exactly why a GPU is a good fit for training and a strange fit for generation.

Figure 01

Stage 01 / 08

Figure 1 — The chain you cannot break. Each decode step needs the token produced by the step before it. The lower timeline contrasts that serial chain against training, where the whole sequence is known in advance and every position is predicted in one pass.

How the callouts are tagged

NUANCE: A subtlety that changes how a result should be read.
TRAP: A conclusion the arithmetic invites but does not support.
APPROXIMATION: A number that is estimated, vendor-quoted or model-derived rather than measured.
ASIDE: A worked example or sanity check. Skippable.

NUANCE · what is and is not serial The dependency is over sampled tokens within one sequence. Independent sequences have no dependency on each other at all, which is why batching across requests is the single most powerful lever in serving. Within a sequence, the only escapes are methods that guess future tokens and verify them cheaply — speculative decoding, covered in §15 — and those do not remove the chain, they shorten it probabilistically.

Two machines, one set of weights

Before deriving anything, it is worth laying the two workloads side by side, because almost every design decision downstream is a consequence of a row in this table.

**Table 1 — The same model under two regimes.** Nothing in the left column is a property of the architecture; every entry is a property of how the architecture is used.
Property	Training	Inference
Work arrives as	curated batches, offline	requests, online, unpredictably
Sequence handling	whole sequences, padded/packed	prompt at once, then one token at a time
Passes per step	forward + backward (+ optimizer)	forward only
FLOPs per parameter per token	≈ 6	≈ 2
Persistent state	weights, grads, optimizer moments	weights, plus a KV cache per request
State growth	fixed for the whole run	grows every single token
Dominant kernel shape	large GEMMs	prefill: large GEMMs · decode: GEMVs
Typical bound	compute, with communication phases	prefill: compute · decode: bandwidth
Optimised for	throughput (tokens/s/$)	throughput and per-request latency
Failure looks like	a slow epoch	a violated latency target
Batch size is	a hyperparameter you choose	an emergent property of arrivals

Two rows deserve immediate comment. The FLOPs per parameter per token row is the familiar 6 versus 2: the backward pass computes gradients with respect to both inputs and weights, costing roughly twice the forward pass, so training is about 3× the arithmetic of inference per token. That makes inference sound cheap. It is not, and the reason is the row below it.

State growth. In training, memory is allocated once and stays flat: the same parameters, gradients and moments occupy the same bytes at step 1 and step 100,000. In inference, every request drags a data structure behind it that grows linearly with every token it generates, is unique to that request, cannot be shared, and must be read in full on every subsequent step. That structure is the KV cache, and by §8 it will be clear that it — not the weights — is what actually limits a serving deployment.

The last row is not a joke. It is the reason inference systems research looks so different from training systems research. A training job that runs 10% slow finishes 10% later. A serving deployment that runs 10% slow may drop below a latency target and become worth nothing at all, which is why §13 spends real effort defining metrics before optimising anything.

Figure 02

Stage 01 / 07

Figure 2 — A flat step versus a growing one. Training repeats an identical step with an identical footprint. A request runs one wide compute-bound phase, then a long tail of narrow bandwidth-bound phases whose state grows monotonically. Bars are schematic proportions, not measured timings.

Prefill and decode

Serving one request means running the model in two distinct modes, and almost every serving system in existence is organised around that split. It is worth deriving rather than asserting.

A request arrives with a prompt of N_p tokens and asks for N_o generated tokens. The prompt tokens are all known simultaneously — there is no dependency among them, because they were given to you. So the first pass can process the entire prompt at once, exactly as training does. That is prefill. Afterwards, each new token depends on the last, so every subsequent pass handles exactly one position. That is decode.

Work in each phase W_prefill ≈ 2·P·N_p + 2·L·N_p²·d (linear layers, then causal attention)
W_{decode step} ≈ 2·P + 4·L·N·d (one token against N cached positions)

The quadratic term in prefill is the familiar N² of attention, halved by causality. It is smaller than people expect at ordinary context lengths: the two terms are equal when N_p = P/(L·d), which for an 8B model with L = 32 and d = 4096 is about 53,000 tokens. Below that, prefill cost is dominated by the dense linear layers and is essentially linear in prompt length; above it, attention takes over and prefill becomes genuinely quadratic.

The shapes are what matter

Take any single linear layer, weights of shape d × d_out. During prefill it sees a matrix of N_p activation rows; during decode it sees one row. That is the same weight matrix in two completely different GEMMs:

One layer, two regimes · d = d_out = 4096, bf16, N_p = 8192 prefill: (8192 × 4096)(4096 × 4096) → I = 2N_pdd_o / [s(N_pd + dd_o + N_pd_o)] = 1638 FLOP/byte
decode: (1 × 4096)(4096 × 4096) → I ≈ 2/s = 1.0 FLOP/byte

Against an H100's machine balance of 295 FLOP/byte, prefill sits comfortably in compute-bound territory and decode sits at the extreme left edge of the roofline, at roughly 0.3% of it. Same weights. Same kernel, even — cuBLAS will happily run both. The only thing that changed is the M dimension of the matrix multiply, and a matrix multiply's arithmetic intensity is governed by its smallest dimension.

Figure 03

Stage 01 / 08

Figure 3 — The M dimension collapses. The weight matrix is identical in both regimes and is streamed from HBM in both regimes. In prefill it is amortised over thousands of activation rows; in decode, over one. Intensities are the single-layer compulsory accounting at the HBM boundary.

Putting numbers on it

Take an 8B model in bf16 on one H100, an 8,192-token prompt, and count both phases end to end.

Prefill W = 2(8.03×10⁹)(8192) + 2(32)(8192²)(4096) = 1.32×10¹⁴ + 1.76×10¹³ = 149 TFLOP
at ~45% of bf16 peak → ≈ 335 ms, i.e. 41 µs per prompt token

One decode step, at 8k context W = 2(8.03×10⁹) + 4(32)(8192)(4096) = 1.61×10¹⁰ + 4.29×10⁹ = 20.4 GFLOP
Q = 16.1 GB (weights) + 1.07 GB (KV) = 17.2 GB → t = Q/β = 5.1 ms
achieved rate = 20.4 GFLOP / 5.1 ms = 4.0 TFLOP/s = 0.40% of peak

Read those two blocks together. Prefill runs the machine at nearly half its rated arithmetic throughput. Decode runs it at four tenths of one percent. The per-token cost differs by a factor of 125, and the model, the precision, the kernel library and the GPU are identical throughout. Every serving optimisation in the rest of this article is an attempt to close some part of that gap.

TRAP · "decode is slow because it is a small kernel" It is tempting to blame launch overhead or poor occupancy. Those exist, but they are second-order. Decode is slow because a decode step must read every parameter in the model to produce one token, and it would still be slow if kernel launches were free and every SM were perfectly occupied. The bound is t ≥ Q/β, and Q is the size of the model. You cannot optimise your way underneath it; you can only make Q smaller (quantization, §14) or make each read produce more tokens (batching §11, speculation §15).

NUANCE · the two phases interfere Because prefill saturates the tensor cores and decode saturates HBM, mixing them in one batch is not free even though it looks like it should be: the prefill attention kernels issue large KV reads that contend with the latency-critical reads of the decode requests sharing the device. This is the observation behind chunked prefill (break long prompts into pieces that fit alongside decode work) and behind prefill/decode disaggregation (run them on different GPUs entirely). §19 returns to it.

Deriving the KV cache

The KV cache is usually introduced as a trick. It is not a trick; it is forced. Start from the naive implementation and the cache falls out of a redundancy so blatant that the only surprising thing is that it needs pointing out.

Suppose you have no cache. At generation step t you hold a sequence of t tokens, you run it through the model, you read off the distribution at the last position, and you sample. Then you append and run t+1 tokens through the model. Now ask what the second pass computed that the first pass had already computed.

The redundancy

In a causal transformer, the key and value vectors at position i are produced by projecting the hidden state at position i:

k_i^(ℓ) = W_K^(ℓ) h_i^(ℓ) v_i^(ℓ) = W_V^(ℓ) h_i^(ℓ)

and the causal mask guarantees that h_i^(ℓ) depends only on x₁…x_i. Appending a token at position t+1 cannot change any x_j for j ≤ t. Therefore every k_i and v_i for i ≤ t is bit-for-bit identical between the two passes. The second forward pass recomputed t sets of key/value projections and got exactly the same answers.

So store them. That single decision converts the per-step cost from "re-run the prefix" to "run one token", and it is the reason interactive generation is possible at all.

Cost of generating N_o tokens after a prompt of N_p (linear-layer terms only) without cache: Σ_{t=1..N_o} 2P(N_p + t) = 2P·(N_oN_p + N_o²/2) — quadratic in output length
with cache: 2P·N_p + 2P·N_o — linear

For an 8B model, a 1,000-token prompt and 500 generated tokens, that is 1.0 × 10¹⁶ FLOP against 2.4 × 10¹³ — a factor of 417×. And the ratio grows without bound as generations get longer, because one side is quadratic and the other is not.

Figure 04

Stage 01 / 10

Figure 4 — The redundancy that forces the cache. Red cells are key/value projections being recomputed from scratch; blue cells are cached and simply read. Causality guarantees the two are numerically identical, which is what makes the substitution exact rather than an approximation.

NUANCE · the cache does not remove attention's cost Caching eliminates recomputation of k and v. It does not eliminate the attention itself: at step t the query must still be compared against all t cached keys, so the attention term stays O(N) per step and the total generation cost stays O(N²) in that term. What collapses is the far larger linear-layer term, which was O(N²) and becomes O(N). At very long contexts the surviving attention term reasserts itself, which is the whole motivation for §7.

The KV memory formula

Count the bytes. At each layer, each token contributes one key vector and one value vector, each of width n_kv · d_head. Multiply through:

Exact KV cache size M_KV = 2 · B · N · L · n_kv · d_head · s

2 — one K and one V · B — concurrent sequences · N — context length
L — layers · n_kv — key/value heads · d_head — head dimension · s — bytes per element

Two things about this expression deserve emphasis. First, d_model does not appear — only n_kv·d_head, which equals d_model only under full multi-head attention. Second, the expression is linear in every one of its terms, so there is no regime in which it becomes benign; there are only regimes in which you have not hit the wall yet.

Factor out the per-token cost, since that is the number worth memorising for any model you work with:

bytes per token = 2 · L · n_kv · d_head · s

**Table 2 — KV cache per token, and what that becomes.** bf16 cache (s = 2), single sequence. Per-token figures are exact binary sizes; totals are decimal GB to match bandwidth units. Note that the 7B MHA model carries four times the cache of an 8B GQA model, at a smaller parameter count.
Model	L	n_kv	d_head	per token	@ 8k	@ 32k	@ 128k
Llama-2-7B (MHA)	32	32	128	512 KiB	4.29 GB	17.2 GB	68.7 GB
Llama-3-8B (GQA)	32	8	128	128 KiB	1.07 GB	4.29 GB	17.2 GB
Mistral-7B (GQA)	32	8	128	128 KiB	1.07 GB	4.29 GB	17.2 GB
Llama-3-70B (GQA)	80	8	128	320 KiB	2.68 GB	10.7 GB	42.9 GB
7B backbone, full MHA	32	32	128	512 KiB	4.29 GB	17.2 GB	68.7 GB

The 128k column is the one to stare at. A single Llama-2-7B-shaped sequence at full context needs 68.7 GB of KV cache — more than five times the model's own bf16 weights, and more than an 80 GB H100 has left after loading them. One request. Not a batch.

Figure 05

Stage 01 / 07

Figure 5 — Linear in everything, which is the problem. Left: cache size against context for four model shapes at batch 1. Right: what remains of an 80 GB card after the weights, converted into a maximum number of resident sequences. Capacity, not bandwidth, is usually what caps the batch.

APPROXIMATION · what this formula leaves out Real allocators do not achieve the theoretical size. A contiguous per-request buffer sized for the maximum possible context wastes everything the request does not use; measured waste in pre-paging systems was severe. Paged allocation (a block table mapping logical positions to fixed-size physical blocks, as in vLLM's PagedAttention) reduces the waste to at most one partly-filled block per sequence, at the cost of an indirection in the attention kernel. Assume a few percent overhead on top of the formula, not zero.

Lab 1 · KV cache calculator

Pick a model shape, a context length and a cache precision, and watch what happens to an 80 GB card. The bar is the whole device; the curve is how many sequences you can hold resident as context grows. The point of the lab is to develop the reflex of computing this before choosing a deployment, rather than discovering it from an out-of-memory error.

Lab 01 · interactive

Model shape

Context length 8192

Concurrent sequences 8

Cache precision

Weight precision

Accelerator

KV per token—

KV per sequence—

KV total—

weights—

device used—

max concurrency—

KV read / decode step—

—

Lab 1. Capacity accounting only — no allocator overhead, no activation workspace, no fragmentation. Real engines reserve a few GB beyond this, so treat the concurrency figure as a ceiling. Weight precision changes the weight bar but not the cache; cache precision changes the cache but not the weights.

MHA, MQA, GQA, MLA

Everything in the cache formula is fixed by the architecture except one term, and that term is where all the engineering has gone. You cannot reduce L without changing the model's depth, you cannot reduce d_head much without hurting attention, and N and B are the workload. But n_kv is free to differ from the number of query heads, because nothing in the attention mechanism requires a one-to-one pairing.

Let n_q be the query heads, n_kv the key/value heads, and g = n_q/n_kv the group size. Then:

Three points on one axis MHA g = 1     cache ∝ n_q        every query head has its own K, V
GQA 1 < g < n_q cache ∝ n_q/g     groups of g query heads share one K, V
MQA g = n_q cache ∝ 1          all query heads share one K, V

The memory effect is obvious: dividing n_kv by g divides the cache by g. The bandwidth effect is the one worth deriving, because it is the same statement viewed from the roofline. During decode, attention reads the whole cache and does a fixed amount of arithmetic per cached element. Per layer:

W = 4·N·d_model (query against N keys, then values)
Q = 2·s·N·n_kv·d_head = 2·s·N·d_model/g
I_{attn, decode} = 4Nd / (2sNd/g) = 2g / s

So the group size is the arithmetic intensity of decode-time attention, up to the constant 2/s. Under full MHA with bf16, g = 1 gives I = 1 FLOP/byte — the worst number in the whole model. Llama-3-8B, with 32 query heads over 8 key/value heads, has g = 4 and I = 4; the 70B, with 64 query heads over the same 8, has g = 8 and I = 8. MQA over 32 query heads would give I = 32. Every one of those is still far short of the ridge at 295, but each factor of g is a real factor of g in both bytes stored and bytes moved.

Crucially, this holds independently of batch size. Batching amortises weight reads across requests because all requests share the weights. No two requests share a KV cache, so batching does nothing for this term at all. That asymmetry is the single most important structural fact in inference serving, and §11 makes it quantitative.

MLA: buying bytes with FLOPs

Multi-head latent attention takes a different route. Instead of storing K and V, it stores a single low-rank latent vector c_t of width d_c per token per layer, and reconstructs the full K and V from it at attention time — with the up-projection folded into the query and output matrices so the reconstruction is nearly free. A small decoupled positional key of width d_h^R is cached alongside it, because rotary embeddings do not commute with the low-rank factorisation.

MLA cache per token = L · (d_c + d_h^R) · s versus GQA's 2 · L · n_kv · d_head · s

With DeepSeek-V2's d_c = 512 and d_h^R = 64, that is 576 elements per token per layer against 2,048 for a GQA-8 model of the same width — roughly 3.6× smaller again, on top of what GQA already saved. Read on the roofline, MLA is the same move FlashAttention made: spend arithmetic, which was idle, to buy back bytes, which were the constraint.

Figure 06

Stage 01 / 08

Figure 6 — One term, four choices. Eight query heads shown. As key/value heads are shared, the cache bar shrinks by exactly the group size and decode-time attention intensity rises by exactly the same factor. The quality axis is qualitative and drawn from the GQA paper's reported trend, not measured here.

TRAP · reading g as a free win Raising g is not costless. Fewer distinct key/value projections is a genuine reduction in model capacity, and the original multi-query work reported both quality degradation and training instability at the extreme. GQA exists precisely because the extremes are bad: it interpolates, and the published result is that a moderate group size recovers close to MHA quality. If you are choosing n_kv for a model you are training, this is an architecture decision with a quality cost, not a serving flag.

NUANCE · full MHA has an inference bill that shows up late A backbone trained with n_kv = n_q carries four to eight times the cache of a contemporary GQA model of the same size. That is invisible during training — the KV cache does not exist there — and becomes the dominant serving constraint the day the model ships. If a model is intended to be served at long context, n_kv is one of the few architecture parameters whose deployment cost should be priced in before pretraining starts, not after.

The inference memory ledger

Write down everything resident on the device during serving. There are four terms, and the interesting thing about them is how differently they behave from their training counterparts.

Inference memory M_inference = M_weights + M_KV + M_activations + M_workspace

M_weights = P · s_w. One copy. No master weights, no gradients, no optimizer moments. This is the entire reason a model that needs a multi-node cluster to train can be served from a single card.

M_KV = 2·B·N·L·n_kv·d_head·s_c. Derived in §5. Unbounded in B·N and unique per request.

M_activations. During decode this is almost nothing: each layer's transients are O(B·d_model) and are freed as the layer completes, so even at batch 256 with a 14k-wide FFN the peak is single-digit megabytes. The one term that can surprise you is the logits tensor, B·V·s, which for a 128k vocabulary at batch 256 in fp32 is 131 MB — manageable, but worth knowing about, and a real problem during prefill if the head is applied to every prompt position instead of only the last one.

M_workspace. Everything the engine itself needs: CUDA graph captures, communication buffers, cuBLAS scratch, the allocator's fragmentation, the block table for paged attention. It is usually one to three gigabytes and it is usually the reason your carefully computed configuration does not quite fit.

Against the training ledger

Set the two side by side for a 7B model, mixed-precision Adam, single device:

**Table 3 — Where the bytes go.** 7B parameters. Training assumes bf16 compute weights, fp32 master weights, fp32 Adam moments, unsharded. Inference assumes bf16 weights and a bf16 cache at 8k context.
Term	Training	Inference	Behaviour
compute weights (bf16)	14 GB	14 GB	fixed both sides
master weights (fp32)	28 GB	—	training only
gradients	14–28 GB	—	training only
Adam moments (m, v)	56 GB	—	training only
activations / checkpoints	10–60 GB	< 1 GB	no backward pass to feed
KV cache	—	1.07 GB × B	grows every token, per request
workspace	1–3 GB	1–3 GB	similar
total	≈ 125–190 GB, constant	≈ 15 GB + 1.07 GB per sequence	fixed vs unbounded

Training's ledger is roughly nine times the weights and it never moves. Inference's ledger starts at roughly one times the weights and then grows without limit in a direction you do not control, because it grows with how much your users type and how much your model says back.

The crossover

Set M_KV = M_weights and solve for the total number of cached tokens across all resident requests:

Where the cache overtakes the model (B·N)_× = P · s_w / (2·L·n_kv·d_head·s_c)

Llama-3-8B, bf16 both: 16.1 GB / 128 KiB = ≈ 122,500 cached tokens
7B full MHA, bf16 both: 13.5 GB / 512 KiB = ≈ 25,700 cached tokens

For the GQA model that is fifteen concurrent 8k conversations. For the full-MHA model it is three. Past that point, more than half of the memory on the device — and, as §10 will show, more than half of the bytes moved on every decode step — belongs to the cache rather than the model.

Figure 07

Stage 01 / 08

Figure 7 — One term is not like the others. Three of the four inference terms are constants. The fourth is a function of your traffic. The right panel sweeps concurrency at 8k context and marks the point where the cache passes the weights.

ASIDE · the prefill logits trap A naive implementation applies the LM head to every position of the prompt, producing an N_p × V logits tensor. At 8k tokens and a 128k vocabulary in fp32 that is 4.2 GB, materialised for no reason — during prefill you only need the distribution at the final position. Every serious engine slices the hidden states before the head. The exception is when the caller asked for per-token logprobs, or when prompt-logprob scoring is enabled, in which case the tensor is real and you should budget for it.

Lab 2 · The serving planner

This is the lab to keep. It puts the capacity accounting of §8 together with the bandwidth accounting that §10 is about to derive, and produces the three numbers a deployment is actually judged on: whether it fits, how long the first token takes, and how long every token after it takes. Everything is a roofline lower bound, so read the times as floors.

Lab 02 · interactive

Model

Prompt tokens 2048

Output tokens 256

Batch 16

Tensor parallel 1

Weights

KV cache

Accelerator

fits?—

TTFT—

TPOT—

E2E latency—

output tok/s, all seqs—

per-user tok/s—

KV share of step bytes—

decode intensity—

—

Lab 2. Model output, not measurement. Prefill assumes 45% of dense peak; decode is a pure bandwidth bound plus a tensor-parallel collective cost modelled at 6 µs of latency per all-reduce, two per layer. Real engines land 20–50% above these floors. Tensor parallelism beyond n_kv cannot shard the cache further and is flagged when it happens.

Why decode is bandwidth-bound

The claim is now easy to state and easy to prove. A decode step must read every parameter in the model, because every layer is used, and it produces one token per sequence. Count both sides for a single sequence:

One decode step, batch 1 Q = s_w·P (all weights) + 2·L·n_kv·d_head·s_c·N (the cache)
W = 2·P (linear layers) + 4·L·N·d (attention)
I = W/Q ≈ 2/s_w = 1 FLOP/byte for bf16

One FLOP per byte, against a machine that demands 295. The arithmetic units will be idle 99.7% of the time and there is no kernel that fixes it, because the ratio is a property of the computation and not of the implementation. Which gives the most useful formula in inference engineering:

The decode wall t_step ≥ (s_w·P + B·M_KV/seq) / β ⇒ per-user tokens/s ≤ β / (s_w·P + B·M_KV/seq)

Everything on the right is known before you write a line of code. An 8B model in bf16 is 16.1 GB; on an H100's 3.35 TB/s that is 4.8 ms per step, so 209 tokens per second is the ceiling — with free arithmetic, a perfect kernel, no cache and no overhead. Measured single-stream numbers land in the 100–140 range, which is a respectable 50–65% of a bound nobody can beat.

Now read the formula for what it licenses and what it forbids. Halving s_w halves the step time; that is weight quantization, and it works. Doubling β halves the step time; that is better hardware or more of it. Reducing the FLOP count does nothing whatsoever, which is why sparsity and low-rank methods so often fail to deliver decode speedups: the weights still have to be read.

Figure 08

Stage 01 / 07

Figure 8 — A ceiling you can compute on paper. Curves are β/(s·P): the single-stream token rate if arithmetic were free. Every real system sits below them. The horizontal band marks roughly where generation stops feeling instantaneous to a reader.

NUANCE · the wall moves as the conversation grows The formula has two terms and only the first is constant. At batch 1 and a short prompt, the cache is a rounding error and the step time is essentially s·P/β. By 128k tokens on an 8B model the cache is 17.2 GB against 16.1 GB of weights, so the step has more than doubled and the model has not changed. Per-token latency that degrades over the course of a long conversation is not a bug in the engine; it is this term.

ASIDE · sanity-checking a published number Whenever someone quotes a tokens-per-second figure, divide the model's byte size by it and compare against the device's bandwidth. If the implied bandwidth exceeds the hardware's spec, either the model is quantised more aggressively than stated, the number is aggregate across a batch rather than per-user, or several GPUs are involved. This check takes ten seconds and catches most confused benchmarks.

What batching actually buys

Since decode is bandwidth-bound and the bytes are mostly weights, the obvious move is to make one weight read serve more than one token. Run B independent sequences in lockstep: each step reads the weights once and multiplies them against B activation vectors instead of one.

Weights only W = 2·P·B Q = s_w·P I = 2B / s_w

Intensity is linear in batch size, and setting it equal to the machine balance gives the batch at which decode would stop being bandwidth-bound:

B* = s_w · π / (2β) = s_w·I* / 2 → H100 bf16: 295 · A100 bf16: 153 · L40S bf16: 419

Below B*, additional requests are close to free: the weights were going to be read regardless, so aggregate throughput rises nearly linearly while per-token latency barely moves. That flat-then-rising shape is the entire economic case for continuous batching, and B* tells you where to aim. Note also that fp8 does not move it — halving s_w and doubling π cancel exactly.

Except the cache does not amortise

Every sequence owns a private KV cache. Nothing is shared, so that term scales with B in the numerator and the denominator. Put both terms in:

The honest intensity of a decode step I(B) = B·(2P + 4·L·N·d) / (s_w·P + B·2·L·n_kv·d_head·s_c·N)

Take B → ∞. The weight term is a constant in the denominator and vanishes:

The batching ceiling I_∞(N) = (2P + 4·L·N·d) / (2·L·n_kv·d_head·s_c·N) = 2P/M_KV/seq + 2g/s_c

and as N → ∞ as well: I_∞∞ = 2g / s_c

This is the result worth taking away from the section. No amount of batching makes long-context decode compute-bound. As context grows, the intensity of a decode step converges to 2g/s_c — the group size and the cache precision, and nothing else. For a GQA-4 model with a bf16 cache that limit is 4 FLOP/byte, or 1.4% of an H100's roof, and you can reach it with a batch of ten thousand.

Concretely, for Llama-3-8B with a bf16 cache:

**Table 4 — Batching returns, by context length.** Llama-3-8B, bf16 weights and cache, one H100. I_∞ is the arithmetic intensity at infinite batch; the throughput ceiling is β/M_KV/seq, which is what the aggregate token rate converges to. B_cross is where cache bytes overtake weight bytes in a step.
Context	KV / sequence	B_cross	I_∞	ceiling as % of peak	throughput ceiling
2,048	0.27 GB	60	63.8	21.6%	12,500 tok/s
8,192	1.07 GB	15	19.0	6.4%	3,120 tok/s
32,768	4.29 GB	3.7	7.7	2.6%	780 tok/s
131,072	17.2 GB	0.9	4.9	1.7%	195 tok/s

The last row is bleak and it is arithmetic, not pessimism: at 128k context a single H100 cannot exceed roughly 195 output tokens per second in aggregate across every user it is serving, because that is the bandwidth divided by the size of one cache. Long-context serving is a fundamentally different economic proposition from short-context serving, and this row is why.

Figure 09

Stage 01 / 08

Figure 9 — Shared weights, private caches. The blue block is read once no matter how many sequences are in flight. The red blocks are read once each. That asymmetry is why the intensity curve bends over instead of continuing to the ridge.

TRAP · quoting B* = 295 as a serving target B* is derived from the weight term alone. It answers "at what batch would the weight read become compute-bound", which is a real question, but it is not the question "what batch should I run". The cache term usually makes the point moot: at 8k context on an 8B model the cache overtakes the weights at batch 15 and the device runs out of memory at batch 57, so B* = 295 is unreachable in both directions at once. Compute B_cross and the capacity limit; treat B* as an upper bound that tells you the arithmetic is not the problem.

NUANCE · latency and throughput point in opposite directions Below B_cross, adding requests raises aggregate throughput while barely touching per-token latency — that region is a genuine free lunch and continuous batching exists to keep you in it. Above it, every added request lengthens every other request's step. There is no batch size that optimises both, which is exactly why serving systems are configured against an explicit latency target rather than against maximum throughput.

Lab 3 · The batching explorer

Three limits govern how large a batch you can usefully run, and they move independently: the arithmetic limit B*, the point where cache bytes overtake weight bytes, and the point where you run out of memory. Sweep context length and cache precision and watch them reorder themselves.

Lab 03 · interactive

Model

Context 8192

KV precision

Accelerator

Tensor parallel 1

B* (arithmetic)—

B_cross (KV = weights)—

B_max (capacity)—

I at B_max—

I ceiling (B → ∞)—

throughput ceiling—

—

Lab 3. Pure roofline bounds with no scheduling model, no preemption and no overhead. The capacity limit assumes 2 GB of workspace and bf16 weights.

Metrics from first principles

Because the two phases have different costs and different consumers, inference has more meaningful latency metrics than training has, and confusing them is the most common way benchmark comparisons go wrong. Define them against a single request's timeline.

Time to first token

TTFT is arrival to first emitted token. It is not "prefill time" — it also contains queueing delay, scheduling delay, tokenisation, and, in a disaggregated deployment, the transfer of the KV cache from the prefill node to the decode node.

TTFT = t_queue + t_schedule + t_prefill ( + t_{KV transfer} )

Under light load, t_prefill dominates and TTFT scales with prompt length. Under heavy load, t_queue dominates and TTFT tells you about your scheduler rather than your model. A benchmark that reports TTFT at batch 1 is reporting the first thing; a production dashboard is nearly always reporting the second.

Time per output token

TPOT — also called inter-token latency, ITL, or time-between-tokens — is the steady-state decode step time from §10, and it is what determines whether text appears to stream smoothly. It is roughly constant for a given batch and context, which is exactly why it is the metric a scheduler can hold to a target.

Composition E2E = TTFT + (N_o − 1) · TPOT

The split matters because the two halves are optimised by opposite means. TTFT is a compute problem: throw more FLOPs at it, chunk it, cache the prefix. TPOT is a bandwidth problem: shrink the bytes, shard them across more HBM, or generate more tokens per weight read. A system tuned for one can be actively bad at the other, which is the entire premise of prefill/decode disaggregation.

Throughput, and why it is not the goal

Output tokens per second across all users is the number that determines cost per token. It is maximised by running the largest batch that fits — which, per §11, maximally degrades TPOT for every user in it. So a serving system optimised purely for throughput produces the worst possible experience while looking excellent on a spreadsheet.

The metric that resolves this is goodput: the rate of requests that complete while meeting their latency targets. A request that finishes with a 4-second TTFT against a 1-second SLO contributes to throughput and contributes nothing to goodput. Optimising goodput is what forces a scheduler to leave capacity on the table, and it is the correct objective for essentially every interactive deployment.

Little's Law, which always applies concurrency = arrival rate × average time in system

If requests arrive at 20 per second and each spends 4 seconds in the system, there are 80 in flight on average — and 80 caches resident. This is the bridge between the queueing picture and the memory picture, and it is how you size a deployment: pick an SLO, derive the average time in system, multiply by the arrival rate, and check that many caches against §8.

Figure 10

Stage 01 / 08

Figure 10 — Throughput and goodput disagree. The last two stages run the same offered load at two batch sizes. The larger batch moves more tokens per second and satisfies fewer requests, because every request's TPOT rose above the target at once.

NUANCE · report percentiles, and report both phases Mean TPOT hides the thing users notice. When a scheduler preempts a request to admit a new prefill, that request sees one very long inter-token gap while the rest look fine; the mean barely moves and the p99 doubles. Any serving comparison that quotes a single number per system, without saying whether it is p50 or p99, at what load, at what prompt and output length, and whether TTFT and TPOT were measured under the same run, is not comparable to anything.

Quantization as a ledger edit

Every quantization scheme is best understood as an edit to one line of the byte ledger, and its effect follows mechanically from which line it edits. There are three lines available.

Weight-only

Store weights at 8 or 4 bits, dequantize to bf16 in registers immediately before the matrix multiply. The GEMM itself is unchanged; only s_w changes. From the decode wall:

t_step = (s_w·P + B·M_KV/seq) / β → halving s_w halves the weight term, exactly

An 8B model at 4 bits is 4 GB rather than 16.1: 1.2 ms per step rather than 4.8, a 4× improvement in the ceiling. This is why weight-only quantization is the highest-leverage single change available for low-batch decode, and why it is the default for local and single-user serving.

And it does essentially nothing for prefill. Prefill is compute-bound at I ≈ 1638; reducing its bytes moves the point further right along a curve that has already gone flat. Worse, the dequantization is real arithmetic added to a phase that was already the constraint, so heavily weight-quantized models frequently prefill slower than their bf16 originals. If your workload is long prompts and short answers, weight-only quantization can be a net loss.

Weights and activations

Quantize both sides so the tensor cores run in the narrow format natively — fp8 on Hopper and later, int8 more broadly. Now π doubles as well as s halving, so both phases improve: prefill gets a genuinely faster GEMM and decode gets fewer bytes. The cost is that activations have outliers and are much harder to quantize than weights, which is why this arrived later and needs more care.

The cache

The third line is the one people reach for last and should often reach for first. From §11, the ceiling on decode arithmetic intensity at long context is 2g/s_c — the cache precision sits in the denominator. Halving s_c therefore does three things at once:

What an fp8 cache buys capacity:        2× the resident sequences at the same context
bandwidth:     half the cache bytes read per decode step
intensity:       the ceiling 2g/s_c doubles, so batching works twice as far

For any deployment whose step bytes are more than half cache — which per §8 is anything past a modest concurrency — cache quantization is strictly the higher-leverage knob, and it is often left at bf16 while people agonise over weight formats.

Figure 11

Stage 01 / 07

Figure 11 — Three knobs, three different effects. Decode time tracks weight bytes; prefill time tracks the arithmetic format; capacity tracks cache bytes. The one row that never improves is prefill under weight-only quantization, where the dequantization cost is added to the binding resource.

APPROXIMATION · the quality axis is not modelled here Every number in this section is a bytes-and-bandwidth claim. None of it says anything about whether the quantized model is as good. Weight-only 4-bit with a decent group size is usually close to lossless on general benchmarks and can degrade sharply on long-context retrieval and on reasoning traces; fp8 activations need per-tensor or per-channel scaling to survive outliers; low-bit KV caches degrade attention over long spans, where the oldest entries have been quantized the longest. Treat the speedups here as the upper half of a trade whose lower half you must measure on your own evaluations.

Speculative decoding

§1 argued that the serial chain cannot be broken. It can, however, be gambled on — and the roofline explains exactly why the gamble is worth taking.

The observation is this. A decode step reads the entire model to produce one token, and during that read the tensor cores sit at 0.34% utilisation. Producing several tokens from the same read would cost almost nothing extra, because the extra cost is arithmetic and arithmetic is the resource you are wasting. The problem is that you do not know what the next several tokens are.

So guess. Run a cheap draft — a small model, a few extra prediction heads, or even an n-gram lookup over the prompt — to propose γ candidate tokens. Then run the real model once on all γ+1 positions and check, in parallel, which proposals it would have made itself. Accept the longest agreeing prefix, discard the rest, and emit a correct token at the first disagreement.

Why verification is nearly free weight bytes read:    s_w·P   — unchanged, γ or no γ
cache bytes read:    unchanged — the same cache, queried from γ+1 rows
arithmetic:          ×(γ+1)   — on units that were 99.7% idle

With a proper rejection-sampling acceptance rule, the output distribution is provably identical to standard sampling from the target model. This is not an approximation: it is the same model, producing the same distribution, in fewer forward passes.

How much faster

Let α be the probability that a proposed token is accepted. Acceptance stops at the first rejection, so the expected number of tokens emitted per target-model pass is a truncated geometric sum:

Expected tokens per verification E[tokens] = (1 − α^γ+1) / (1 − α)

speed-up ≈ E[tokens] / (1 + γ·c) where c is the draft's cost relative to one target step

At α = 0.8 with γ = 4 that is 3.36 tokens per pass; with a draft costing 5% of a target step the net speed-up is about 2.8×. Acceptance rate is the whole ballgame — it depends on how well the draft mimics the target, and it varies enormously by domain. Code and structured output accept extremely well; open-ended creative text much less so.

Where it stops working

Speculation spends idle arithmetic. When the arithmetic is no longer idle, the trade inverts. Verifying γ+1 positions for each of B sequences puts B(γ+1) rows through every GEMM, so the effective intensity is 2B(γ+1)/s_w and the step becomes compute-bound once

B(γ+1) > B* = s_w·π/(2β) → on an H100 in bf16, B(γ+1) > 295

At batch 8 with γ = 4 you are at 40 — deep in free territory. At batch 64 with γ = 4 you are at 320, past the ridge, and every rejected token is now real wasted time. This is why speculative decoding is transformative for single-user and low-concurrency serving and frequently disabled on high-throughput deployments: the resource it arbitrages has already been sold.

Figure 12

Stage 01 / 08

Figure 12 — Buying tokens with idle FLOPs. The weight read is identical whether you verify one position or five. The curves show expected tokens per verification against acceptance rate, and the point at which B(γ+1) crosses the ridge and the arbitrage closes.

NUANCE · the draft does not have to be a model Anything that produces plausible continuations cheaply will do. Extra prediction heads on the target model itself avoid running a second network at all; tree-structured drafts verify several candidate continuations at once and raise the effective acceptance rate; and for tasks with heavy verbatim repetition — retrieval-augmented answering, code editing, summarisation of a provided document — a plain n-gram lookup into the prompt is nearly free and accepts remarkably often. Match the draft to the redundancy structure of your workload rather than reaching for a small model by default.

Distributed inference

Training uses many GPUs because the work is enormous. Inference uses many GPUs for two quite different reasons: the model may not fit, and — more interestingly — more GPUs means more aggregate bandwidth, which per §10 is the only thing that makes decode faster. The four ways of splitting the work behave very differently.

Tensor parallelism

Shard every weight matrix across n devices: attention by heads, the FFN by its hidden dimension. Each device holds P/n parameters and, if n ≤ n_kv, n_kv/n key-value heads and hence 1/n of the cache. Each device reads 1/n of the bytes, all of them concurrently, so the aggregate bandwidth really is nβ:

t_step ≈ (s_w·P + B·M_KV/seq) / (n·β) + 2L · t_all-reduce

The first term scales perfectly. The second does not. Each layer needs two all-reduces — one after attention's output projection, one after the FFN's — and during decode the payload is B·d_model·s bytes, which is tiny: at batch 1 and d = 4096 in bf16, 8 kilobytes. A transfer that small is entirely latency, not bandwidth. So the collective cost is roughly 2L times a fixed per-collective latency, and it does not shrink as you add devices.

8B model, batch 1, H100 + NVLink, ~6 µs per collective n = 1: 4.79 ms · n = 2: 2.40 + 0.38 = 2.78 ms (1.7×)
n = 4: 1.20 + 0.38 = 1.58 ms (3.0×) · n = 8: 0.60 + 0.38 = 0.98 ms (4.9×)

Eight GPUs buy about five times the token rate, and the missing three are sitting in 64 small collectives. That is why custom low-latency all-reduce kernels exist in every serious inference engine, and why tensor parallelism across nodes — where the per-collective latency is much higher — is generally a bad idea for decode even when it is fine for prefill.

One more constraint: you cannot shard fewer than one key-value head per rank. Past n = n_kv the cache must be replicated, so memory stops improving even though the weights keep sharding. A GQA-8 model on 16-way tensor parallelism holds two copies of every cache.

Pipeline parallelism

Split by layers instead. Device 1 holds layers 1–20, device 2 holds 21–40, and so on. This fits large models across nodes with almost no communication — one activation tensor per boundary, point-to-point — but it does nothing for single-token latency: the token still has to traverse every stage in order, and now it pays a network hop between each. Pipelining helps throughput only when there are enough independent requests in flight to keep every stage busy, and decode's per-step granularity makes the bubbles harder to fill than in training. Use it to fit, not to accelerate.

Replicas

Run n independent copies. Throughput scales exactly linearly, latency is unchanged, and nothing is shared — including prefix caches, which is a real cost if your traffic has common prefixes. This is the right answer whenever the model fits and you need capacity rather than speed.

Expert parallelism

A mixture-of-experts layer routes each token to k of E experts. At batch 1 the decode step reads only the shared parameters plus the k active experts, so the bytes moved scale with active parameters, not total. That is a genuinely large win: a model with eight times the parameters can decode at close to the speed of its active subset.

It erodes with batch, and quickly. Different tokens route to different experts, so the number of distinct experts the step must read is

E[distinct experts] = E · (1 − (1 − k/E)^B) (uniform routing)

For E = 8 and k = 2 that is 2 experts at batch 1, 5.5 at batch 4, and 7.9 at batch 16. By moderate concurrency you are reading nearly every expert anyway and the byte advantage is gone — the FLOP advantage remains, but FLOPs were not the constraint. MoE is therefore most valuable exactly where speculative decoding is: low-concurrency, latency-sensitive serving.

Figure 13

Stage 01 / 08

Figure 13 — More bandwidth, minus the collectives. Tensor parallelism divides the byte term perfectly and adds a constant that does not divide. The dashed line is ideal scaling; the gap between it and the solid line is 2L all-reduce latencies. Collective cost is modelled at 6 µs each and is the least certain number in this figure.

NUANCE · tensor parallelism is also a memory lever, up to a point Sharding the cache across ranks is often the real reason to raise n, not the bandwidth. But the cache only shards while n ≤ n_kv. Beyond that, engines replicate the key-value heads and per-device cache memory stops falling, so a GQA-8 model gets no capacity benefit from going past 8-way — while still paying the extra collectives. Check n against n_kv before assuming more devices means more room.

Vision-language inference

Everything so far applies unchanged to a vision-language model, because a VLM is a language model with a longer prompt. What changes is how that prompt gets long, how expensive it is to construct, and how unpredictably it varies between requests. Those three changes are enough to make VLM serving a distinctly harder problem than text serving, and they are worth working through concretely.

The pipeline, for a modern crop-and-pool architecture:

From pixels to the first token image → decode & resize → aspect-ratio tiling → ViT encoder → connector → visual tokens
→ concatenated with the text prompt → LLM prefill → decode

Take a concrete configuration of the kind used by current open VLMs: a SigLIP-2 So400m encoder at patch 14 and 384×384 input, which emits a 27 × 27 grid of 729 tokens per tile; dynamic tiling that splits a high-resolution image into an aspect-ratio-aware grid of up to twelve 384×384 tiles plus a global thumbnail; a pooling connector that compresses 4:1; and a 7B language backbone.

Where the tokens come from

13 tiles × 729 tokens = 9,477 patch tokens
after 4:1 pooling: ≈ 2,370 visual tokens

A user who types twenty words and attaches one photograph has just submitted a 2,400-token prompt. The connector is not a detail here — without the 4:1 pooling the prompt would be 9,500 tokens and every downstream cost would be four times larger. Choosing the connector's compression ratio is one of the highest-leverage serving decisions in a VLM, and it is made at training time.

Where the time goes

Three costs stack before the LLM has done anything.

TTFT budget, one image, 7B backbone, H100 preprocess (JPEG decode, resize, 13 crops)    ≈ 15 ms on CPU
ViT: 2·P_vit·9477 + 4·L_vit·729²·d_vit·13 ≈ 8.4 TFLOP   ≈ 21 ms
LLM prefill over 2,400 tokens ≈ 36.6 TFLOP             ≈ 82 ms
TTFT ≈ 118 ms   versus ≈ 1 ms for the same question with no image

The preprocessing line is the one that surprises people. It is pure CPU work — image decode, resize, normalisation, thirteen crops — it does not appear in any GPU profile, and in a poorly pipelined server it is serial with everything else. On a busy node with many concurrent image requests it can become the actual bottleneck while every GPU dashboard shows headroom.

Where the memory goes

Visual tokens are ordinary tokens once they reach the backbone, which means they occupy KV cache exactly like text, forever, for the life of the request. With a full-MHA 7B backbone at 512 KiB per token:

one image ≈ 2,370 tokens × 512 KiB = 1.24 GB of KV cache
the accompanying 30-token question = 15.7 MB
the image costs about 80× the text it came with

An 80 GB card holding a 7B model in bf16 has roughly 63 GB free, so it can hold about 50 single-image conversations — and about twelve if users attach four images each. Swap the backbone to GQA-8 and the same card holds four times as many. This is the concrete form of the argument in §7: the attention shape you choose at pretraining time sets the serving economics of every image your model will ever see.

Where the scheduling gets hard

Text prompts vary in length; image prompts vary in structure. A wide panorama, a tall screenshot and a small square thumbnail produce different tile grids — anywhere from 1 to 13 tiles — so the visual token count within a single batch can vary by more than an order of magnitude. Three consequences follow:

Ragged vision batches. The ViT wants uniform-shape tiles, which tiling conveniently provides, but the number of tiles per request varies, so the encoder's batch is ragged. Padding to the maximum wastes up to 92% of the work in a batch with one large and several small images.

Unpredictable prefill. TTFT is no longer a function of prompt tokens, because prompt tokens are not known until the image has been decoded and tiled. A scheduler making admission decisions on request metadata is estimating blind unless preprocessing runs first.

Cache pressure arrives in lumps. A text request adds tokens one at a time; an image request adds 2,370 at once. Admission control that works well for text can thrash badly when a burst of image requests each demand a gigabyte of cache simultaneously.

On the other hand, VLM traffic has a redundancy text traffic lacks: users ask several questions about the same image. Prefix caching over the visual tokens turns the second and subsequent questions into ordinary short-prompt requests, cutting TTFT by an order of magnitude. If your workload has that shape, prefix caching is the single highest-return thing to implement.

Figure 14

Stage 01 / 10

Figure 14 — One photograph, 2,370 tokens. Counters follow a single request through the pipeline. The vision encoder is compute-bound and fixed-cost per image; the LLM prefill that follows is four times larger again; and the cache it leaves behind outlives both. Configuration is a So400m-class encoder at 729 tokens per tile with 4:1 pooling and a full-MHA 7B backbone.

NUANCE · the vision encoder is a fixed cost with no decode phase The ViT runs exactly once per image and never again. It has no KV cache, no autoregressive dependency and no decode analogue — it is a pure compute-bound feedforward pass, and it batches beautifully across tiles. That makes it the one part of a VLM request that behaves like a training workload, and the one part that a prefill/decode split naturally puts on the prefill side. In a disaggregated deployment the encoder belongs with prefill, and its output — not the pixels — is what should be cached.

TRAP · counting an image as "one token" of cost Pricing, rate limiting and context accounting for VLMs frequently treat an image as a small fixed cost. It is not: at these settings one photograph is 2,370 tokens of prefill compute and 1.24 GB of cache, and both scale with the image's resolution through the tile count. A four-image request at full resolution can exceed the cache footprint of a hundred text conversations. Any capacity model that does not compute visual tokens from the tiling rule will be wrong by an order of magnitude in the direction that hurts.

Lab 4 · The image budget

The tiling rule, the pooling ratio and the backbone's attention shape are all decided long before a request arrives, and between them they set what an image costs to serve. This lab makes that chain explicit: change a training-time decision on the left, watch the serving bill on the right.

Lab 04 · interactive

Tiles per image 13

Vision encoder

Connector pooling

Images per request 1

Language backbone

Text tokens 30

KV cache

Accelerator

patch tokens—

visual tokens—

prompt tokens—

TTFT—

vision share of TTFT—

KV per request—

concurrent conversations—

vs. text-only prompt—

—

Lab 4. Model output, not measurement. Vision encoder FLOPs are 2·P_vit·N + 4·L_vit·N²·d_vit per tile at bidirectional attention, run at 40% of dense peak; LLM prefill at 45%; preprocessing modelled at 1.2 ms per tile of CPU work, which is the crudest number here and the one most sensitive to your image pipeline. Concurrency assumes the backbone in bf16 plus 2 GB of workspace.

ASIDE · try the two extremes Set pooling to 1:1 and tiles to 25 and you get a request whose prompt is nearly 20,000 tokens before the user has typed anything — a configuration that some early high-resolution VLMs genuinely shipped, and which explains why their serving costs were brutal. Then set pooling to 16:1 with a 224-pixel encoder and watch TTFT fall by more than an order of magnitude. Everything between those poles is a trade against how much fine visual detail survives the connector, which is a model-quality question this lab cannot answer for you.

The scheduling problem

Every number so far assumed a batch that already exists: fixed size, uniform prompt lengths, all sequences starting and finishing together. No real deployment looks like that. Requests arrive at unpredictable times, with prompts spanning four orders of magnitude, generating outputs whose length is not known until generation stops, under latency targets that differ by product surface. The gap between the batch we analysed and the traffic that actually arrives is where most of the practical performance of a serving system lives.

Consider eight requests with realistic variation — prompts from 128 to 8,192 tokens, outputs from 20 to 500.

Static batching wastes most of the machine

The obvious implementation collects a batch, runs it to completion, and returns. But the batch cannot return until its slowest member finishes, so a sequence that stops after 20 tokens leaves its slot occupied and idle for another 480 steps. For the eight requests above, the occupied-slot utilisation is

Σ output tokens / (B × max output) = 1,180 / (8 × 500) = 29.5%

Nearly three quarters of the batch dimension — the dimension §11 showed is the only source of arithmetic intensity — is spent computing padding.

Continuous batching

The fix is to schedule at the granularity of the iteration rather than the request. After every decode step, evict finished sequences and admit waiting ones into their slots. The batch composition changes constantly; the batch size stays near its maximum. This one change typically multiplies throughput several-fold on real traffic, and it is the single most important idea in modern serving.

Prefill and decode fight each other

Admitting a new request means running its prefill, and prefill is a large compute-bound operation. If an 8,192-token prompt is prefilled as one unit while thirty sequences are mid-generation, all thirty see a decode step that took 5 ms suddenly take 340. Users experience this as the stream visibly hitching. There are two established answers:

Chunked prefill splits the prompt into pieces and attaches each piece to a decode batch, so every iteration does a bounded amount of prefill work alongside its decodes. Inter-token latency stays smooth at some cost to raw prefill throughput, and the batch's arithmetic intensity rises usefully because the prefill chunk brings FLOPs to a step that had none.

Disaggregation goes further and puts prefill and decode on separate pools of GPUs, transferring the KV cache between them. This is a direct consequence of §3: the two phases have opposite bottlenecks, so they want different hardware, different parallelism and different batch sizes. Prefill nodes can run compute-optimised configurations; decode nodes can be chosen purely for bandwidth. The cost is a cache transfer on every request, which is why this only pays above a certain scale.

Memory is allocated in pages, not blocks

A sequence's final length is unknown when it starts, so a naive allocator reserves the maximum context up front — and then most of it goes unused, because most generations are short. Reserving 128k of cache for a request that produces 40 tokens wastes gigabytes. Paged allocation solves this the way virtual memory does: the cache lives in fixed-size blocks, allocated on demand, referenced through a per-sequence block table. Fragmentation drops from most of the cache to under a block per sequence, and the same mechanism makes copy-on-write prefix sharing nearly free.

What is left over

Preemption, when memory runs out mid-generation and a sequence must be evicted — either swapped to host memory or discarded and recomputed later, where recomputation is often cheaper because prefill is fast and PCIe is not. Prefix caching, which turns shared system prompts and repeated images into cache hits. Priority classes and admission control, which decide whose latency degrades when the machine is full. And the goodput question from §13: a scheduler maximising tokens per second will happily produce tokens nobody is waiting for any more.

Figure 15

Stage 01 / 08

Figure 15 — The same eight requests, four schedulers. Blue is prefill, green is decode, hatched red is a slot held but doing nothing. Static batching runs the whole batch at the pace of its longest generation; continuous batching refills slots the moment they free; chunked prefill trades a little prefill throughput for a flat inter-token latency; disaggregation gives each phase its own hardware.

TRAP · benchmarking with uniform requests A load test that sends identical prompts with identical output lengths measures none of this. Static batching looks fine, continuous batching looks unnecessary, chunked prefill looks like pure overhead, and paging looks like a solution to a problem you do not have. Every one of those conclusions reverses under real traffic. If you take one methodological point from this article, take this: benchmark with the length distribution and arrival pattern of your actual workload, and report tail latencies, not means.

Which is where this article stops and the next one begins. Everything here has been about a single request or a single batch on a single machine, reasoned from bytes and FLOPs. The scheduling problem sketched above is a different kind of problem — queueing, admission, fairness, SLO negotiation, cache placement across a fleet — and it deserves its own treatment. That is the subject of the next piece: The Serving Playbook.

What should survive from this one is smaller and more portable. Prefill is compute-bound and decode is memory-bound, and almost every surprising fact about inference performance is a corollary of that sentence. When something is slower than you expect, work out which of the two phases you are in, count the bytes that phase must move, divide by bandwidth, and compare with what you measured. The gap is where the answer is.

References

The derivations here are all first-principles arithmetic, but the ideas are not mine. These are the primary sources, grouped roughly in the order the article uses them.

Attention and its variants

Attention Is All You Need — Vaswani et al., 2017. The architecture, and the source of the causal-masking structure that makes the KV cache possible.
Fast Transformer Decoding: One Write-Head is All You Need — Shazeer, 2019. Multi-query attention, and the earliest clear statement that decode is memory-bandwidth-bound.
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Ainslie et al., 2023. The interpolation between MHA and MQA that almost every current model uses.
DeepSeek-V2 — DeepSeek-AI, 2024. Multi-head latent attention: caching a low-rank latent instead of keys and values.
FlashAttention — Dao et al., 2022. Why the attention kernel's cost is about memory traffic rather than FLOPs, and the tiling that fixes it.

The roofline and the hardware

Roofline: An Insightful Visual Performance Model for Multicore Architectures — Williams, Waterman & Patterson, 2009. The model everything in this article is built on.
Data Movement Is All You Need — Ivanov et al., 2020. A careful accounting of where transformer time actually goes.
Efficiently Scaling Transformer Inference — Pope et al., 2022. The partitioning analysis behind most of §16, including why decode's collectives dominate.

Quantization and speculation

GPTQ — Frantar et al., 2022. Accurate post-training quantization of weights to 4 bits.
AWQ: Activation-aware Weight Quantization — Lin et al., 2023.
SmoothQuant — Xiao et al., 2022. The activation-outlier problem, and why W8A8 is harder than W8A16.
Fast Inference from Transformers via Speculative Decoding — Leviathan, Kalman & Matias, 2022. The acceptance-rate expectation used in §15.
Accelerating Large Language Model Decoding with Speculative Sampling — Chen et al., 2023. The concurrent derivation, with the distribution-preservation proof.
Medusa — Cai et al., 2024. Speculation without a separate draft model.

Serving systems

Orca: A Distributed Serving System for Transformer-Based Generative Models — Yu et al., OSDI 2022. Iteration-level scheduling — continuous batching.
Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023. vLLM, and the paged cache of §19.
SARATHI — Agrawal et al., 2023, and Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve, OSDI 2024. Chunked prefill.
DistServe — Zhong et al., 2024. Disaggregating prefill and decode, with the goodput argument.
Splitwise — Patel et al., 2023. The same split, motivated by phase-specific hardware.
Mooncake — Qin et al., 2024. A KV-cache-centric architecture with disaggregated cache storage.

Vision-language models

SigLIP 2 — Tschannen et al., 2025. The encoder family and resolutions used in §17's token accounting.
Molmo and PixMo — Deitke et al., 2024. Crop-based high-resolution processing with a pooling connector.
CogVLM and InternVL 1.5 — dynamic tiling and its effect on visual token counts.
Learning Transferable Visual Models From Natural Language Supervision — Radford et al., 2021. CLIP, for the encoder lineage.