The Serving Playbook

One request, one GPU

The field's direction in one sentence. Serving evolved from filling idle arithmetic units, to managing per-request state, to separating workloads that want different hardware and schedules.

Put a single 8B-parameter model on a single H100 and send it one request. The prompt goes in, tokens come out, the user is happy. Now open a profiler and look at what the GPU is actually doing. The answer is: almost nothing.

Decoding is autoregressive. To produce token t, you need token t−1, so every token costs one full forward pass through the network. In that forward pass, the model reads every weight exactly once and performs roughly 2Ψ floating-point operations, where Ψ is the parameter count. For an 8B model in bf16 that is 16 GB read from HBM and 16 GFLOPs of arithmetic.

Divide one by the other and you get the arithmetic intensity of a decode step: 1 FLOP per byte. An H100 SXM delivers about 3.35 TB/s of HBM bandwidth and about 989 TFLOP/s of dense bf16 tensor-core throughput, so its balance point — the intensity at which the two run out simultaneously — sits near 295 FLOPs per byte. A batch-1 decode step is roughly 295× below that line. You are paying for a machine that can do a thousand trillion operations per second and using it to move memory around.

That last line is the whole reason serving systems exist. The weight read is a fixed cost per iteration. It does not care whether one sequence or two hundred sequences are riding along. Every sequence you add to the batch converts a matrix–vector product into a slightly taller matrix–matrix product, which the tensor cores were built for and which costs no extra weight traffic at all. Batching is not an optimization you bolt on later. It is the only way the hardware makes sense.

Prefill behaves in the opposite way. A 2,000-token prompt does 2Ψ×2000 FLOPs against the same single weight read, which lands it far to the right of the balance point. Prefill is compute-bound and already efficient at batch 1; decode is memory-bound and catastrophically inefficient at batch 1. Almost every technique in this article exists to move decode rightward on that axis without wrecking latency.

Static batching, and why it disappoints

The obvious first move: collect requests until you have B of them, pad them into a rectangle, run the batch to completion, return the results, repeat. This is how you batch a ResNet, and it works there because every image takes exactly one forward pass.

Generation is not like that. Four things go wrong at once.

• Arrival jitter. The batch cannot start until it is full, so the first request to arrive pays the wait time of the last. Under light load this dominates time-to-first-token entirely.
• Length heterogeneity. Output lengths in real traffic vary by an order of magnitude — a yes/no answer and a code file share a queue. The batch runs for as long as the longest member.
• Padding. Rectangular tensors mean short prompts carry dead columns through every layer.
• Slot squatting. A sequence that emits its end-of-sequence token on iteration 4 keeps occupying its batch slot, its KV allocation and its share of every GEMM until iteration 24.

The last one is the expensive one, and it is worth staring at.

The batch you paid for is not the batch you get. Effective batch size decays monotonically from B to 1 over the life of the group, so the arithmetic intensity you carefully bought in section 1 leaks away iteration by iteration, and newly arrived requests sit in the queue watching it happen.

Continuous batching

The fix comes from asking a slightly rude question:

Why is the batch a property of the request group at all?

The model does not know what a "batch" is. Each iteration it receives some set of token positions, multiplies them by weights, and returns logits. Nothing requires that set to be the same from one iteration to the next. So make it not be.

This is iteration-level scheduling, introduced by Orca and now standard everywhere. At every decode step the scheduler assembles a fresh batch from whatever is currently runnable. A sequence that finishes is evicted immediately and its slot is refilled from the waiting queue on the very next iteration. Nobody waits for anybody.

There is a subtlety that makes this harder than it sounds. You cannot simply stack sequences of different lengths into a rectangular tensor. Orca's answer is selective batching: the operations that are position-independent — every linear layer, the MLP, the normalizations — get a flattened, ragged token dimension, so a batch is just [total_tokens, d_model] with no padding whatsoever. Attention, which must not mix sequences, is handled per-sequence using cumulative-length offsets, which is precisely the interface a variable-length attention kernel exposes.

Once the batch is ragged, prefill tokens and decode tokens stop being different species. A prefill contributes many rows, a decode contributes one, and the GEMM does not care. That observation is what makes sections 8 and 9 possible.

The scheduler you are actually building

At this point the interesting object is no longer the model. It is a small resource allocator that runs once per iteration and answers one question: which sequences run in the next forward pass?

It holds sequences in a handful of states — waiting, running, and (when memory runs out) preempted — and it allocates four scarce things:

• A token budget, the maximum number of token positions in one iteration. This bounds iteration time, and therefore bounds the inter-token latency of every sequence in flight.
• Sequence slots, a cap on concurrent sequences, which bounds per-sequence bookkeeping and attention kernel overhead.
• KV blocks, the real constraint, discussed next.
• Priority, which decides who gets the above when they are contended.

Admission is not just "is there a free slot". A sequence admitted now will need one more KV block every block_size steps for as long as it lives, so schedulers hold a watermark of free blocks and refuse to admit a request that would leave the running set unable to grow. When the watermark is breached anyway — because everyone kept generating — something has to be evicted.

Eviction has two flavours and the choice is a genuine trade. Recompute throws the KV away and re-prefills the sequence when it is readmitted: free to evict, costs one prefill later. Swap copies KV to host memory over PCIe and copies it back: costs bandwidth in both directions but preserves the work. Recompute is the usual default, though not for the reason people give — section 21 does the arithmetic and finds the swap is actually cheaper in raw time. What decides it is when the memory comes back, not how much the round trip costs.

The KV allocation problem

Attention needs the keys and values of every previous position, so a serving system stores them. The size is exact and unforgiving:

On an 80 GB H100 holding 16 GB of weights and a few GB of activations and workspace, you have perhaps 60 GB for KV. That is sixty full-length sequences, or six hundred short ones. The KV cache, not the model, is what determines how many users you can serve.

Now the naive implementation. You do not know how long a response will be, so you reserve the maximum: one contiguous buffer of max_model_len tokens per sequence, allocated at admission. Two kinds of waste follow immediately.

Internal waste. If the average response is 400 tokens and you reserved 8,192, then 95% of every reservation is untouched. Concurrency drops by the same factor, and with it your arithmetic intensity, and with that your throughput. The waste is not in memory, it is in the batch size you could have had.

External waste. If you instead reserve per-request maxima of different sizes, the free pool fragments. After a few hundred admissions and completions you can have 12 GB free and still be unable to place a 3 GB request, because the free space is in pieces. The PagedAttention authors measured existing systems wasting 60–80% of their KV memory this way.

An operating systems person reading the previous paragraph is already impatient. We solved this in 1962.

Paged KV cache

The only reason a sequence's KV needed to be contiguous is that the attention kernel indexed it with a stride. That is a kernel convention, not a mathematical requirement. Position i of a sequence has no opinion about where position i−1 physically lives.

So do what virtual memory does. Cut KV into fixed-size blocks of, say, 16 tokens. Give each sequence a block table mapping its logical block indices to physical block numbers. Let the attention kernel gather through that table. This is PagedAttention, and its properties fall straight out of the analogy:

• External fragmentation goes to zero. Every free block is interchangeable, so a free block is always usable.
• Internal waste is bounded by at most block_size − 1 tokens in the last block of each sequence — under a kilobyte per sequence rather than a gigabyte.
• Allocation becomes lazy. You allocate a block when you fill one, not when you admit a request, so you never pay for tokens that were never generated.
• Sharing becomes free. Two sequences whose prefixes are identical can point at the same physical blocks. Reference-count them, and copy-on-write only the partially filled block where they diverge.

That last property is the one that keeps paying dividends. Parallel sampling, beam search, and — most importantly — shared system prompts all reduce to "several block tables, one set of physical blocks".

The cost is real but small: an extra indirection in the hottest kernel in the system, and a gather pattern that is less friendly to the memory system than a flat stride. This is why paged attention kernels are hand-written rather than expressed as a library call, and why block size is a tuning parameter — small blocks reduce waste and increase table overhead, large blocks do the reverse. Sixteen is a common compromise.

Only now is it fair to say the name: this is what vLLM is. The scheduler and the block manager are the product; the model is a plugin.

Prefix caching

Look at real traffic and you will find it is not a set of independent prompts. It is a set of prompts that share enormous amounts of text: a system prompt repeated across every request, a few-shot block, a retrieved document reused across a hundred questions, and in chat, the entire conversation so far resent on every turn.

Because attention is causal, the KV state of a prefix depends only on the tokens in that prefix. It is a pure function of the token sequence. So if you have computed it once, recomputing it is arithmetic you are choosing to pay for.

Two implementations dominate, and they are the same idea at different granularities. SGLang's RadixAttention keeps a radix tree over token sequences whose nodes own KV blocks, so a new request walks the tree to find its longest cached prefix. vLLM hashes each block's contents together with the hash of all preceding blocks, which is the same tree flattened into a hash table.

Eviction has one structural rule: never evict a node that has live children, because a child's KV is only valid if its ancestors are resident. So eviction proceeds leaf-first, ordered by last use. And a cache hit is only a hit if the request lands on the replica that holds the blocks — which turns prefix caching into a routing problem the moment you have more than one GPU. Section 18 comes back to this.

Chunked prefill

Continuous batching gave us the right to mix sequences freely in an iteration. Now watch what happens when one of them is a 16,000-token prompt.

That prefill is a single enormous forward pass. For an 8B model it is roughly 2 × 8e9 × 16384 ≈ 262 TFLOPs, which at a realistic 400 TFLOP/s of effective throughput takes about 650 ms. During those 650 ms, every one of the sixty sequences that were happily decoding at 20 ms per token produces nothing. Their inter-token latency spikes by 30×, and the user watching a stream sees the text freeze.

The prefill is not doing anything wrong. It is simply an indivisible unit of work that is far too large relative to the iteration cadence everyone else depends on. So divide it.

Chunked prefill, from Sarathi, splits a long prompt into fixed-size chunks of C tokens and processes one chunk per iteration, piggybacking the running decodes into the same batch. The token budget becomes the scheduler's master clock: every iteration contains exactly C tokens, some prefill and some decode, so every iteration takes about the same time.

Nothing is recomputed — chunk k attends to the KV that chunks 0 through k−1 already wrote, which the paged cache is happy to provide. What you do pay is weight traffic: each chunk re-reads the model weights, so a 512-token chunk has arithmetic intensity 512 instead of 16,384. Above the balance point that is invisible; below it, prefill efficiency collapses. This gives the tuning rule directly:

Sarathi-Serve calls the second half of this stall-free batching: the scheduler never admits a unit of work that would push an iteration past the latency target, so decodes are never stalled by an admission decision.

Prefill and decode want different machines

Put the two phases on a roofline and the tension becomes geometric rather than rhetorical.

The optimistic reading is that these two workloads are complementary: one is starved of arithmetic, the other of bandwidth, so mixing them should fill both pipes. The real benefit is narrower than that, and worth stating precisely.

When a prefill chunk and forty decodes share an iteration, the weights are read from HBM once and used by both. The decodes ride along on a memory transfer that was going to happen anyway. That is a genuine and large win, and it is the actual mechanism behind piggybacking.

What does not happen is free parallelism inside the GPU. Both phases need the same weight matrices and the same SMs, and a kernel that is compute-bound does not leave idle tensor cores for someone else to use. So mixing raises efficiency, but it also means the decodes in that iteration finish no sooner than the whole iteration does. Their TPOT is set by the chunk size, which is why chunk size is a latency knob and not just a throughput knob.

Attention is the other spoiler. Prefill attention and decode attention are different kernels with different shapes — a square causal problem versus a long thin one — and they do not fuse into a single call. In practice they run as separate kernels within the iteration, which is why mixed batches sometimes lose the benefits of graph capture (section 15).

Scheduling policy

With the mechanism settled, the remaining question is who goes first. Serving inherits every classical scheduling problem plus one new handicap: you do not know how long a job will run. Output length is decided by the model, one token at a time.

FIFO is the default and is defensible. It is starvation-free and predictable, and under continuous batching the head-of-line penalty is bounded by the token budget rather than by the longest sequence. Its failure mode is a burst of very long prompts arriving before a burst of short ones.

Shortest-job-first would minimise mean latency, and cannot be implemented honestly. Systems that approximate it predict output length from the prompt, which works until it does not, and a mispredicted long job that has been scheduled as short must be preempted rather than merely regretted.

Priority classes are what production actually uses: interactive traffic above batch traffic, paying tiers above free tiers. The mechanism is preemption, and the danger is starvation of the low class under sustained load, so priority is nearly always paired with an aging term or a reserved share of the token budget.

Fairness in this setting is not requests per second, because requests are wildly unequal. The right unit is tokens: a fair queue counts the token positions each client has consumed, prefill and decode alike, and services whoever is furthest behind their share. This is the only policy that survives a single client sending 128k-token prompts.

Token-budget scheduling deserves its own line because it is the mechanism underneath all of the above. Once every iteration is capped at C tokens, iteration time is roughly constant, and inter-token latency becomes a scheduler guarantee rather than an emergent property. Policy then operates on how the budget is divided rather than on whether iterations are the right size.

Finally, SLO-aware admission. A serving system that accepts everything under overload will miss every deadline instead of some of them. Queueing a request, shedding it, or routing it to a lower-cost configuration are all better outcomes than degrading the whole running set. Section 23 makes that argument quantitatively.

Disaggregating prefill and decode

Chunking made the two phases coexist. It did not make them stop wanting different hardware configurations.

Prefill is compute-bound, benefits from high tensor parallelism to cut latency, and needs almost no KV capacity per request — it produces KV rather than accumulating it. Decode is bandwidth-bound, benefits from the largest batch memory will allow, and is dominated by KV capacity. Co-located, you must pick one parallelism degree, one batch policy, and one memory split that serves both badly.

So separate them into pools connected by a KV transfer.

What you buy is independence. The pools scale separately, so you can run eight decode replicas behind two prefill replicas if that is what your prompt-to-output ratio demands. They can use different parallelism degrees, different batch sizes, even different hardware generations — older, cheaper, bandwidth-rich parts are perfectly good decode machines. And most importantly, the two SLOs decouple completely: a prefill surge can no longer perturb anyone's inter-token latency, because prefill is not on the decode machines.

What you pay is the transfer. Using the earlier figure of 128 KiB per token, a 4,000-token prompt hands off 512 MiB of KV. At 400 Gb/s that is roughly 10 ms, which is tolerable against a 500 ms TTFT budget — and it is also entirely hideable, because layer ℓ's KV is final the moment layer ℓ finishes, so the transfer streams behind the remaining layers of prefill. What is not free is everything around it: interconnect provisioning between pools, a router that knows where each request's KV lives, failure handling when a decode node dies holding live state, and the awkward question of which pool owns the prefix cache. Mooncake's answer — treat the KV cache as the primary object and build the cluster around it — is the honest conclusion of this line of thinking.

Distributed serving: scale up or scale out

One GPU is not the interesting case. Four ways to use many, each answering a different question.

Replicas

The simplest and most robust: N independent copies behind a router. Throughput scales linearly, failures are isolated, and there is no collective communication anywhere. It does nothing for per-token latency and nothing for models that do not fit, and each replica's KV capacity is limited by whatever the weights left behind.

Tensor parallelism

Shard each weight matrix across GPUs and all-reduce twice per transformer layer. This is the only technique that makes a single token faster, and it also divides the weights, which frees HBM for KV — a doubly good deal for decode. The catch is the communication profile: a decode all-reduce moves only batch × d_model elements, so it is pure latency, not bandwidth. Small messages, twice per layer, sixty-four times per token. Anything but a fast intra-node fabric turns this into the bottleneck.

Pipeline parallelism

Assign layer ranges to devices and pass activations forward. Communication is one small tensor per stage boundary, which makes it the only parallelism that tolerates slow links gracefully. In serving, the bubble is not a throughput problem in the training sense — continuous batching supplies a steady stream of micro-batches — but it does add latency per stage, and it does not reduce per-GPU weight memory as usefully as TP does for a single request.

Expert parallelism

For mixture-of-experts models, place experts on different devices and route tokens to them with an all-to-all. Weight memory per device drops dramatically for a given total capacity, which is why MoE is attractive for serving in the first place. The problem is that routing is data-dependent: a batch that happens to favour a few experts leaves most devices idle while a couple are swamped, and the all-to-all runs at the speed of the slowest.

Topology awareness

Every choice above assumes GPUs are interchangeable. They are not. Inside an 8-GPU node, NVLink gives each GPU hundreds of gigabytes per second to every other. Between nodes, a fast NIC gives you 400 Gb/s — 50 GB/s — and adds microseconds of latency that the intra-node fabric does not have. That is roughly an order of magnitude in bandwidth and considerably more in small-message latency.

Three rules follow, and they are close to absolute.

Tensor parallel groups stay inside one NVLink domain. TP's all-reduces are small and frequent, exactly the profile that punishes a network with a per-message cost. A TP=16 group spanning two nodes usually performs worse than TP=8 on one, despite having twice the hardware.

Replicas are packed, not striped. A replica that spans nodes inherits the worst link in its group for every collective. Pack each replica onto a single node, then scale out with independent replicas; this also means a node failure costs you one replica rather than degrading many.

Cross-node traffic is the one you choose deliberately. If something must cross the fabric, make it pipeline stage boundaries or disaggregated KV transfers — large, infrequent, latency-tolerant messages that overlap with compute — rather than per-layer all-reduces. And check NIC affinity: a GPU pushing KV through a NIC on a different PCIe root complex loses a surprising amount of the bandwidth you paid for.

For MoE, add a fourth: expert placement is a topology decision. Keeping an all-to-all inside a node, or replicating the handful of consistently hot experts so that routing has a local option, is often worth more than any kernel-level optimization.

Speculative decoding

Return to section 1 one more time. A decode forward pass at batch 1 uses about 0.3% of the machine's arithmetic. The tokens are serial because each depends on the last — but the verification of a guessed token does not depend on anything. If you already had a candidate for tokens t through t+4, you could score all five positions in one forward pass for almost exactly the price of scoring one.

So guess. A cheap draft — a small model, an extra prediction head, or an n-gram lookup over the prompt — proposes γ tokens. The target model runs once over all γ+1 positions and produces the true distribution at each. Then a rejection-sampling rule accepts the longest prefix of the draft that is consistent with the target's distribution and resamples the first rejected position from an adjusted distribution. The output is distributionally identical to ordinary sampling from the target model. This is not an approximation.

The variants differ in where the draft comes from. An independent small model is simple but needs a matching tokenizer and adds a second model to serve. Medusa attaches extra heads to the target model itself and verifies a tree of candidates at once. EAGLE drafts in feature space rather than token space, which raises acceptance substantially for the same draft cost. Prompt lookup skips the model entirely and proposes n-grams copied from the input, which is startlingly effective for summarization, editing and code-completion workloads where the output quotes the input.

The catch is the one the derivation predicts. Speculation spends spare FLOPs, and spare FLOPs only exist at small batch. Once continuous batching has filled the machine, the extra γ+1 positions per sequence are no longer free — they are a multiplicative increase in real compute, paid on every token including the rejected ones. Speculative decoding is a latency technique that costs throughput, and the crossover is a property of your load, not of your model. Good implementations disable it dynamically when the running batch gets large.

CUDA graphs and the CPU you forgot about

A decode iteration for a 32-layer model launches on the order of several hundred kernels: attention, projections, norms, activations, residuals, sampling. Each one is tiny — tens of microseconds of GPU work at small batch. Each one also costs the CPU a few microseconds to launch.

Multiply it out. If the average kernel does 15 µs of work and takes 6 µs to launch, the GPU is idle 40% of the time waiting for its instructions, and no kernel optimization will fix that because the kernels are not the problem. Below a certain batch size, an LLM server is a Python program bottlenecked on Python.

A CUDA graph records the entire dependency DAG of an iteration once and replays it with a single launch. The GPU-side work is unchanged; the CPU-side cost collapses to nearly zero.

The price is rigidity. A captured graph pins tensor shapes and device addresses, so a serving system must: bucket batch sizes and capture a graph per bucket, padding up to the nearest one; preallocate every buffer the iteration touches, including block tables and slot mappings, at fixed maximum sizes; and keep all control flow out of the captured region, which means sampling parameters become tensors rather than Python branches. Prefill, whose shapes change with every prompt, is normally left in eager mode — and this is exactly why a mixed chunked-prefill iteration often falls back to eager execution, giving back some of what chunking won at small batch. Systems resolve this in different ways, including capturing graphs for the decode portion only, or restructuring the scheduler so that CPU work for iteration i+1 overlaps GPU work for iteration i and the launch cost is hidden rather than eliminated.

Kernel fusion, briefly

Fusion is the same argument as batching, applied within an iteration: every kernel boundary is an HBM round trip, and HBM is what you are short of.

• Attention. FlashAttention never materializes the n×n score matrix, tiling the computation so it lives in SRAM. For decode, where the query is a single position and the KV is long, Flash-Decoding splits the reduction along the KV axis instead, which is the only way to get parallelism when there is one query row and a hundred thousand keys.
• Norms and residuals. RMSNorm fused with the residual add saves reading and writing the hidden state twice per block. Small individually, multiplied by 2L per token.
• Activations. SwiGLU fused into the gate and up projections avoids materializing two full intermediates at d_ff width.
• Sampling. The one people forget. Top-k, top-p, penalties and multinomial sampling done on the host require copying logits back and synchronizing, which serializes the CPU against the GPU exactly where you least want it. Fused GPU sampling removes a synchronization point per token, which matters more than its FLOP count suggests.

The pattern: in serving, the enemy is rarely arithmetic. It is memory round trips and synchronization.

Quantized serving

Quantization is usually discussed as a compression technique. In serving it is better understood as three separate levers that happen to share a name, because each targets a different bottleneck.

Weight-only quantization (GPTQ, AWQ, typically 4-bit weights with 16-bit activations) attacks weight bandwidth. Since decode time at small batch is 2Ψ×bits/8 ÷ BW, cutting weights from 16 to 4 bits cuts decode latency by close to 4×. It also frees HBM for KV. But it does nothing for compute, and the dequantization work is real, so as batch size grows and the workload crosses the balance point, the advantage shrinks to nothing and can invert.

Activation quantization (W8A8: SmoothQuant, or FP8 on Hopper) attacks compute. It uses the low-precision tensor-core paths, so it speeds up prefill and large-batch decode — precisely the regime weight-only quantization abandons. The difficulty is activation outliers, which is what SmoothQuant's rescaling between activations and weights is for.

KV quantization attacks the constraint that actually limits concurrency. At long context and high batch, KV dwarfs the weights: sixty sequences of 8k tokens at 128 KiB per token is 60 GB against 16 GB of weights. Halving KV precision doubles the batch you can hold, and also halves the attention memory traffic that we noted in section 1 refuses to amortize. Eight-bit KV is close to free in quality terms. Four-bit needs care — KIVI's observation is that keys have consistent per-channel outliers while values do not, so K wants per-channel quantization and V wants per-token.

Routing across replicas

Now there are twenty replicas and something in front of them. That something is making decisions that matter more than most kernel work.

Load must be measured in tokens, not requests. Least-connections routing assumes requests are comparable; here one request can be 400× another. The useful signals are pending prefill tokens, running sequence count, and KV utilization — all of which the replicas already track and can export.

Cache locality fights load balance. Sending every request that shares a system prompt to one replica maximizes prefix hits and creates a hotspot. The workable policy is a soft one: prefer the replica with the longest matching cached prefix, unless its queue exceeds the fleet average by some margin, in which case pay the recompute. Multi-turn sessions get the same treatment with a stickiness that decays under pressure.

Failure is a cache-coherence problem. When a replica dies, the KV of every in-flight sequence dies with it. Retries on another replica must re-prefill from the original prompt, which means the router has to keep the request, not just a handle to it, and the client-visible effect is a latency spike rather than an error. In disaggregated deployments this gets worse: a decode node holds state that no prefill node can reconstruct without redoing the prompt.

Stragglers are structural. One replica holding a few 128k-token sequences will have a genuinely slower iteration cadence than its neighbours for as long as those sequences live. Percentile latency across the fleet is therefore shaped by the worst-loaded replica, which argues for admission control at the router rather than only inside each replica.

Serving vision–language models

Everything so far assumed the prompt is text and its length is known when the request arrives. A VLM breaks both assumptions, and the consequences reach into every layer of the design.

The prompt length is data, not metadata

Modern VLMs use dynamic resolution: an image is tiled or resized according to its own aspect ratio and detail, so it contributes anywhere from a few dozen to several thousand visual tokens. The scheduler cannot compute a token budget from the request body without running the preprocessor first. Every admission decision therefore depends on work that has not happened yet, which means either you preprocess before admission (adding latency to the critical path) or you admit on an estimate and handle being wrong.

Preprocessing is a CPU service

JPEG decode, resize, normalize and tile are host-side work, and at meaningful QPS they will saturate the CPUs on a GPU node long before the GPUs are busy. This belongs in a separate process pool with its own queue, overlapped with GPU execution, and ideally with decoded tensors cached — otherwise you have built an expensive accelerator that waits on libjpeg.

The encoder is a second model with different tastes

A ViT over a few thousand patches is compute-bound, fixed-shape per tile, and embarrassingly batchable across requests. The language model is memory-bound and ragged. Their ideal batch sizes, ideal parallelism degrees and ideal schedulers disagree. The work ratio between them also swings by two orders of magnitude across requests: a single small image with a 500-token answer is decoder-dominated, while twelve high-resolution tiles with a one-word answer is almost entirely encoder. Running the encoder as its own pool — a small-scale version of section 11's disaggregation — is the natural response, with the connector output shipped to the language pool.

Cache embeddings, not just KV

In a multi-turn conversation about one image, or a workload where the same document page is queried repeatedly, the post-connector embeddings are identical every time. They are keyed by the image content hash plus the preprocessing configuration, and they are far smaller than the KV they produce. This is the cheapest win in VLM serving and the one most often missing.

Prefix caching must hash pixels

This is a correctness trap worth stating loudly. Visual tokens occupy placeholder positions in the token id sequence, and those placeholder ids are identical for every image. A prefix cache that hashes token ids alone will conclude that two requests with completely different images share a prefix, and serve one image's KV to the other's question. The block hash must incorporate the image hash. It is a quiet bug: the outputs remain fluent.

Visual tokens are a scheduling knob

Unlike text, you control how many tokens an image becomes. Resolution selection, tile count, and token merging or pruning in the connector all trade answer quality for prefill cost, per request. Under load, a server can serve the same model at a lower visual token budget rather than queueing — a graceful degradation axis text-only serving simply does not have.

One last scheduling wrinkle: chunked prefill interacts awkwardly with images. The language-model prefill over an image's visual tokens can be chunked normally, but the encoder pass that produced them cannot be split mid-image without changing the result. An image is an indivisible unit of encoder work sitting inside an otherwise perfectly divisible prefill.

Reading a serving system's architecture

Feature lists age badly. Every system eventually grows every feature, and a table of checkmarks tells you almost nothing about how a server will behave at three in the morning under a traffic spike.

What does not change is the bottleneck each project was originally built against. That first choice decides which abstraction sits at the centre of the codebase, and the central abstraction is the expensive thing to reverse. Everything else gets arranged around it.

The five questions

Applied to any new system, these predict behaviour under load better than a benchmark table:

vLLM up close: the allocator is the scheduler

vLLM's bet is stated in section 20 and worth repeating before taking it apart: on a decoding workload the scarce resource is KV memory, and if you build a good enough allocator for it, several things that look like separate features stop being projects and become consequences.

That claim only earns its keep at the level of mechanism. Four places where it does.

Where num_gpu_blocks comes from

Everything downstream — how many sequences run concurrently, how large a batch the GEMMs see, how far right you sit on the roofline from section 1 — is bounded by a single integer computed once at startup. It is worth being able to derive it from memory.

Two details in that arithmetic are more interesting than they look. The weights are measured after loading rather than derived from the parameter count, because quantization formats, LoRA adapters, and layout padding all move the real number. And the activation peak comes from an actual profiling forward pass at the configured maximum batch and sequence length, not from a formula — which is why a misconfigured max_num_batched_tokens shows up as a startup failure rather than as a mysterious out-of-memory four hours into a traffic spike.

The practical consequence is that gpu_memory_utilization is not a safety knob, it is a concurrency knob. Moving it from 0.85 to 0.95 on an 80 GB card hands eight more gigabytes to the KV pool, which at a 4k context is sixteen more concurrent sequences, which is a straightforward multiplier on throughput. The reason you cannot simply set it to 1.0 is that the profiling run cannot see everything: fragmentation inside the allocator, communication buffers, and the odd transient all live in the margin you leave behind.

The block table lives in the hottest kernel in the system

Paging is cheap to describe and expensive to implement, because the indirection lands in exactly the code that runs most often. Understanding why the attention kernel is hand-written starts with the shape it is given.

A decode step gives the kernel one query row per sequence and a KV cache thousands of rows tall. Parallelizing the classical way — over query positions and heads — produces a handful of thread blocks for a machine with 132 streaming multiprocessors. So the parallelism has to come from the KV axis instead: each thread block takes one block of the cache, looks up where it physically lives, computes a partial softmax over its sixteen tokens, and a second pass rescales the partials into the answer. That is the same online-softmax reduction that Flash-Decoding uses, with a pointer chase bolted onto the front.

The pointer chase is what makes the layout fussy. A gather that lands wherever the free list happened to point will not coalesce unless the data inside a block is arranged so that consecutive lanes still read consecutive addresses; the original CUDA implementation reshapes the head dimension into short vectors specifically so that the loads stay wide despite the indirection. This is also why block_size resists a default: internal waste falls with smaller blocks, block-table traffic and gather efficiency both improve with larger ones, and sixteen is where the three curves cross for typical traffic. On workloads dominated by very long generations, thirty-two is often better; on workloads of very short replies it is strictly worse.

Prefix caching as a side effect of the allocator

Here is the payoff for having built the allocator first. Blocks are fixed-size and aligned, so a block's contents are a well-defined thing to name. Name them by hashing — and the reuse machinery is a dictionary.

The chain is the load-bearing part. A block's key is the hash of its own token ids folded together with the hash of every block before it, so a key means this content, at this offset, after this history. Hash the block's own tokens alone and two unrelated conversations containing the same sixteen tokens at the same offset collide, and one user's KV is served against another user's prompt. The output stays perfectly fluent, which is what makes this class of bug so unpleasant.

The second detail worth knowing is that eviction is lazy in a useful way. A block whose refcount falls to zero goes to the tail of a free list but keeps its hash entry; it is only unregistered at the moment it is actually handed to a different sequence. So a prefix can come back from the dead, and a measured hit rate can be higher than the live working set would explain.

Sharing is a reference count

The same mechanism absorbs three features that would otherwise each need their own code path. Parallel sampling with n=8 over a 2,000-token prompt does not copy anything: eight block tables point at the same 125 physical blocks, refcount eight, and only the partially filled final block is copied when the samples diverge. Beam search is the same operation with more forking and pruning. A shared system prompt across unrelated users is the same operation again, discovered by hash rather than declared by the API. Figure 6 shows the copy-on-write moment; the point here is that there is only one moment, reused everywhere.

The memory arithmetic is worth doing once. Naively, eight samples of a 2,000-token prompt cost eight copies of 250 MiB. Shared, they cost 250 MiB plus eight partial blocks — about 4 MiB of divergence. That is the difference between a feature you ration and a feature you leave on.

Preemption, where the folklore is wrong

When the watermark is breached something has to be evicted, and vLLM offers two policies. The received wisdom is that recomputation is cheaper than swapping. Run the numbers and that turns out to be false, for an interesting reason.

A KV byte moves across PCIe far faster than a FLOP regenerates it, and grouped-query attention makes the KV small relative to the model's arithmetic, so on raw time the swap wins by roughly eight times. The ratio narrows for full multi-head models, where KV per token is four times larger, but it does not usually invert.

Recompute is nonetheless the default, and every reason is about timing rather than magnitude. It frees the blocks now, whereas a swap only relieves pressure after the copy lands — and pressure is why you are preempting. It needs no reserved host buffer, so it cannot itself run out. It does not contend for a PCIe link that disaggregated deployments may already be using for KV transfer. And with prefix caching on, the re-prefill frequently hits the cache and costs almost nothing, which collapses the 327 ms to a rounding error. Swapping earns its place in the narrow band where prompts are long, the prefix cache cannot help, and preemption is frequent enough that the recompute bill compounds.

Why the scheduler ends up thin

Notice what the scheduler in this design actually does. It asks the block manager whether a candidate can be admitted, it counts tokens against a budget, and it decides who to preempt. That is nearly all of it — chunked prefill from section 8 is just the token budget applied to a prompt, and prefix reuse happens inside the allocator without the scheduler being told. The difficulty was moved rather than removed: the block manager is where the invariants live, and a bug there is a correctness bug rather than a performance one.

The overhead this design does not address is the one section 22 is about. Once memory is no longer the constraint, the Python that runs between iterations becomes visible, which is why the engine was later restructured to run its core in a separate process with scheduling overlapped against execution — the same conclusion, reached from the other end.

When this is the wrong bet

Paging costs an indirection in the hottest kernel and a tuning parameter that has no universally right value. If your workload is a fixed batch of identical shapes — offline scoring, a benchmark harness, a batch job with uniform prompts — there is no fragmentation to eliminate and no sharing to discover, and a compiled engine with static shapes will simply be faster. Portability is also a deliberate cost: keeping the attention backend pluggable across vendors means declining some of the fusion a single-target compiler would take. The bet pays when your traffic is heterogeneous, bursty, and full of accidental repetition, which is what production traffic mostly is.

SGLang up close: the runtime as a cache

The survey above is deliberately shallow. It is worth going one level down into a single system, and SGLang is the most instructive one to pick, because its design starts from a premise the rest of this article has quietly been assuming away.

That premise: a request is not an independent event. An agent loop issues twenty calls that share a system prompt and a growing scratchpad. A few-shot pipeline sends the same four thousand tokens of examples in front of every query. A tree-of-thought search forks the same context a dozen ways. A chat turn resends the entire conversation. In every one of these, the overwhelming majority of the tokens the server sees have been seen before, often seconds ago, often on the same GPU. Once you take that seriously, the interesting object is no longer the request queue. It is the structure the requests share.

RadixAttention: the four operations

The data structure is a radix tree — a compressed trie — over token id sequences. Each edge carries a span of tokens; each node owns the KV blocks for its span; the path from the root to any node spells a prefix whose KV is resident. Four operations do all the work, and the third one is where the "radix" earns its name.

Match. Walk from the root comparing token ids. The cost is proportional to the length of the match, not to the size of the cache, which matters because the cache is large and the match is what you are about to reuse anyway.

Split. The awkward case: the request agrees with a node for part of its span and then diverges mid-node. The node is cut into a parent holding the agreed prefix and a child holding the rest. This is the operation people are surprised by, and the thing worth internalizing is that no KV moves. The blocks stay exactly where they are; only the tree's edges are rewritten. Splitting is a pointer edit.

Insert. The unmatched suffix becomes a new child of the split point, and that suffix is the only thing prefilled. Twelve new tokens against a hundred resident ones is a typical ratio for a chat turn.

Evict. Every node on the path of a running request carries a reference count and cannot be freed — a sequence is mid-generation and its attention will read those blocks on the next iteration. Eviction is therefore LRU restricted to unpinned leaves, applied recursively, because freeing a leaf may turn its parent into the next candidate. The rule that a node with live children can never go is what keeps the invariant intact: a prefix is only valid if its own prefix is resident.

The tree changes what the scheduler should do

Here is the consequence that a hit-rate metric hides. If the cache is a passive thing you consult, arrival order decides your hit rate, and arrival order is adversarial: alternating prefixes evict each other in exactly the wrong sequence. If the cache is a structure the scheduler can see, the waiting queue can be reordered to group requests that share ancestors, and the same eight requests produce half the prefill work.

There is a second, quieter benefit. Requests batched together that share a prefix are also reading the same KV blocks in the attention kernel, so the shared portion is fetched from HBM once for the batch rather than once per sequence. Recall from section 1 that attention is the part that refuses to amortize with batch size — prefix sharing is one of the few things that makes it amortize anyway.

And the obvious hazard: unbounded reordering starves cold prefixes. In practice the sort is bounded, by a window over the queue, by an aging term, or by a rule that a request may only be deferred so many times. This is the same tension as section 18's routing problem, one level down: cache locality and fairness pull in opposite directions, and something has to arbitrate.

Getting the CPU off the critical path

Once prefix reuse has shortened prefill and continuous batching has shortened iterations, the next thing to become visible is embarrassing: the server's own bookkeeping. Building the next batch, assembling block tables, preparing sampling metadata, and detokenizing the last step's output are all host work, and at a twelve-millisecond iteration they are not hiding behind anything.

The fix is structural rather than clever: prepare iteration i+1 on the CPU while iteration i is executing, and move tokenization and detokenization into separate processes so a slow detokenize can never stall the loop. Note the relationship to section 15 — CUDA graphs attack kernel-launch overhead, this attacks scheduling overhead, and they are the same disease at different altitudes. A system can need both.

Structured output, where most of the tokens are not decisions

SGLang treats constrained generation as a serving problem rather than a sampling detail, and the reason is visible the moment you count. Ask for JSON matching a schema and a large fraction of the output is determined: after the opening brace the next characters must be the first key, and no amount of sampling will change that. The conventional approach masks the logits so only legal tokens survive — correct, but it still runs a full forward pass to produce logits that the grammar was going to override anyway.

Compressing deterministic runs of the finite-state machine into a single edge lets the runtime emit them directly, and the model is only consulted where the grammar leaves a genuine choice. For tool calls and fixed schemas — where the payload is a handful of values inside a lot of punctuation — this removes most of the model calls in the response.

It is not free of sharp edges. A jumped string has to be re-tokenized before it can be appended to the KV cache, and the tokenizer may not split it the way the model would have; implementations handle this by backing off the last few characters of a jump and letting the model resolve the boundary. The mask computation itself is also real work — walking a context-free grammar against a 100k-token vocabulary every step is expensive enough that dedicated engines such as XGrammar exist to precompute which parts of the vocabulary are context-independent and to overlap the rest with GPU execution.

When none of this helps

Worth stating plainly, because it is the honest test of any of these designs. If your traffic is independent one-shot prompts with no shared structure, RadixAttention holds a tree that never matches and cache-aware scheduling sorts a queue with nothing to sort. If your batches are already large, the overlapped scheduler is hiding an overhead that was already hidden. If you never emit structured output, the FSM machinery is dead code. The tree also costs memory: KV held for a prefix that might be reused is KV not available to a running sequence, so the eviction policy is arbitrating between speculative reuse and actual concurrency.

Which is the general lesson, and it applies to every system in the previous section. These are not universal improvements. They are bets about the shape of the workload, and the reason to understand the mechanism rather than the benchmark is so you can tell whether the bet matches yours.

The objective is goodput

Every technique in this article has a knob, and every knob moves throughput and latency in opposite directions. Larger batches, larger chunks, more aggressive admission, more speculation: each one has a setting that maximizes tokens per second and a different setting that keeps users happy. Optimizing throughput alone produces a server that is magnificently efficient at missing deadlines.

So state the objective properly. Fix the service-level objectives first — a time-to-first-token target and an inter-token latency target, both at a percentile, because means hide the failures. Then:

This quantity — goodput — behaves very differently from throughput. Throughput rises with load and then flattens. Goodput rises, peaks, and then falls off a cliff, because past the knee every additional admitted request pushes the whole running set over the latency line and you get paid for none of them.

Reading the article backwards through this lens makes it much shorter. Continuous batching raises the ceiling of the goodput curve. Paged KV and prefix caching move the knee to the right by making concurrency cheaper. Chunked prefill flattens the TPOT tail so the constraint binds later. Disaggregation decouples the two constraints so they stop trading against each other. Speculation buys TTFT and TPOT headroom at low load and must be switched off at high load. Quantization relaxes the memory constraint. Routing prevents one replica from setting the fleet's percentile. Admission control is what keeps you on the left side of the cliff when the knobs run out.

None of it is about making the model faster. All of it is about keeping expensive silicon doing work that someone is still waiting for.

Once the system is assembled, the final question is diagnostic: how do we find what it is actually waiting for?

References

Grouped by topic. Every entry links to its primary source.