Talk Is Not Cheap

Communication is another memory hierarchy

Blog 02 ended with a single uncomfortable fact: an H100 can perform roughly 990 trillion bfloat16 operations per second, and read only 3.35 trillion bytes per second from its own memory. Arithmetic is nearly free. Movement is what you pay for.

That blog stopped at the edge of the die. This one asks the obvious next question. What changes when the data an operation needs is not on this GPU at all?

The answer is: nothing changes conceptually, and everything changes quantitatively. Reaching another GPU is the same kind of event as an HBM access — a request travels somewhere, a response comes back, and the tensor cores idle in the meantime. It is simply further away. So the right mental model is not "memory hierarchy, and then separately, networking". It is one continuous ladder that happens to be built out of several different technologies.

Every rung on that ladder is the same operation — fetch a byte that lives somewhere else — and the only thing that varies is how much it costs. Which gives us the framing for the whole article:

The ladder is not smooth

Before looking at latency, look at bandwidth, because the shape of the ladder is the thing worth internalising. For each rung, divide the tensor cores' 989 TFLOP/s by the bandwidth available. That gives the break-even arithmetic intensity: how many operations you must perform per byte fetched from that level to keep the machine busy.

Read down the last column. The first three rungs span a factor of nine — a gentle slope. Then HBM to NVLink is a 7× step. NVLink to the network is another 9×. These are not gradients, they are cliffs, and almost every design decision in the rest of this article is an attempt to keep heavy traffic above a cliff and send only light traffic over it.

Nuance 01 Arithmetic intensity is per level, not per chip

Often saidAn H100 needs 295 FLOPs per byte to break even.

ActuallyOnly against its own HBM. The break-even ratio is a property of the level, not the chip: roughly 33 FLOP/byte against shared memory, 295 against HBM, 2,200 against NVLink and about 19,800 against the fabric. A kernel that is comfortably compute-bound on HBM can be catastrophically communication-bound the moment its operand has to cross a node boundary. Whenever anyone quotes an arithmetic intensity, ask which level it is measured against.

Notice how uneven the ladder is. The first three rungs span a factor of nine. The step from HBM to NVLink is another 7×. The step from NVLink to the network is another 9×. There is no smooth gradient here — there are cliffs, and the entire discipline of parallelism strategy is the art of arranging your computation so that the heavy traffic stays above the cliffs and only the light traffic goes over them.

This article walks down the whole ladder from first principles. We will build a cost model that predicts collective runtimes to within a factor of two, derive the ring AllReduce and prove it is bandwidth-optimal, work out exactly which collective fires where in data, tensor, pipeline, context and expert parallelism, and finish with the failure modes that make real clusters slower than the arithmetic says they should be.

Why communication costs time

Before reaching for a formula, it is worth asking a naive question and answering it properly: when I send a message from one GPU to another, where does the time actually go?

The answer is not one thing. It is six, and they behave very differently from each other. Some scale with how much data you send. Some are completely indifferent to it. Getting a feel for which is which is most of the intuition you need.

Walk the six once more, because the split between them is the whole point.

Software overhead

Launching the kernel, posting the work request, writing the descriptor. Nothing has physically moved yet. On a modern stack this is frequently the largest single component for small messages.

Injection

The doorbell write that tells the NIC there is work, the NIC's DMA read of the descriptor, and getting the first bit onto the wire.

Serialisation

Pushing n bytes through a link of width B. Takes exactly n/B seconds, and it is the only component in the list that cares how big your message is.

Propagation

Flight time. Signals travel at roughly two thirds of light speed in fibre or copper, which works out to about 5 ns per metre. Thirty metres of cable is 150 ns, each way.

Switching

Every hop has to read a header and choose an output port. Proportional to the number of hops, which is a property of your network topology rather than your message.

Queueing

Waiting behind someone else's traffic. The only component that depends on what the rest of the cluster is doing, and therefore the only one that varies run to run.

Now the observation that organises everything. Group those six by whether they care about n:

• Indifferent to n: software overhead, injection, propagation, switching, queueing. Sum them and you get a fixed toll paid once per message.
• Proportional to n: serialisation, and only serialisation.

A cost with a fixed part and a part proportional to size is an affine function. We did not choose that form for convenience — it fell out of asking where the time goes. Which is where the next section starts.

The α–β model, derived

We just found that message time splits into a part that ignores n and a part proportional to n. Give those two parts names and the model writes itself.

That is the entire model. It is worth being clear about what it is not: it is not an approximation someone chose because it was tractable, and α is not a fudge factor. α is the sum of five real, physically distinct costs we enumerated in §2, and β is the reciprocal of a link width. The model is affine because the underlying process is.

The one number to remember

Set the two terms equal and solve. The message size at which the fixed cost exactly equals the variable cost is

This is the half-power message size, and it earns its name: at n = n_½ you are spending half your time on overhead, so you achieve exactly half the link's peak bandwidth. It cleanly divides the world in two.

• Below n_½ you are in the latency-dominated regime. Sending twice as much data barely costs more. The way to go faster is to send fewer messages, not smaller ones.
• Above n_½ you are in the bandwidth-dominated regime. Time is essentially proportional to bytes, and the only thing that helps is moving less data or finding a fatter pipe.

Almost every tuning decision later in this article — bucket sizes, chunk counts, which collective algorithm to pick — is really a decision about which side of n_½ you want to be on.

The figure lets you feel the crossover directly. Drag the size and distance sliders and watch which band swallows the bar.

One consequence deserves stating on its own line, because it catches people out constantly. Bandwidth-delay product — link rate times round-trip time — is the number of bytes that must be in flight simultaneously to keep a link busy. On a 400 Gb/s link with a 10 µs round trip that is 500 KB. If your messages are smaller than that and you wait for each one, you will use a fraction of the link no matter what the sticker says. This is why RDMA fabrics lean so hard on deep queues and many outstanding operations, and why a collective that issues one small message at a time is a catastrophe.

Representative values

These are the numbers I use for back-of-envelope work on H100 clusters. They are deliberately pessimistic against marketing peak, because achievable bandwidth on collectives is typically 70–85% of link rate once protocol overhead and imperfect overlap are counted.

Stare at the last row. EFA's α is roughly four times InfiniBand's, which pushes n_½ up to half a megabyte. Any collective on an EFA cluster operating on chunks smaller than ~500 KB is running latency-dominated and will not come close to the bandwidth on the invoice. That single fact drives a startling number of practical decisions later on.

One more term, for reductions

Collectives that reduce do arithmetic on the way, so strictly:

On GPUs γ is almost always negligible against β — summing bf16 at 3.35 TB/s of HBM bandwidth is far faster than moving the same bytes over a 50 GB/s NIC. I will carry γ through the derivations for correctness and drop it in the numbers. On CPU clusters, or when the reduction is something expensive, it stops being negligible.

PCIe

PCIe is the substrate everything else sits on. Even in a machine where GPUs talk to each other over NVLink, PCIe is still how the GPU gets its work, how the driver rings its doorbells, and — critically — how data reaches the network card. It carries three distinct kinds of traffic:

• CPU ↔ GPU — kernel launches, parameter uploads, anything staged through host memory.
• GPU ↔ NIC — every byte that leaves the node, on most hardware.
• GPU ↔ GPU — on machines without NVLink, or between GPUs NVLink does not connect.

To reason about any of them you need one structural fact.

PCIe is a tree, not a bus

The name is a fossil. The original PCI really was a bus — a shared set of wires that every device listened on. PCI Express is a switched point-to-point fabric arranged as a tree. At the root sits the root complex, part of the CPU, which is the only thing that talks to memory. Below it are switches, and at the leaves are endpoints: GPUs, NICs, NVMe drives.

That topology decides everything about what is fast. Two devices under the same switch are close. Two devices on opposite sides of the machine are not, and the packets between them have to climb up the tree and back down — possibly through a CPU that has nothing to do with the transfer.

Three practical rules fall straight out of that picture.

Peer-to-peer is a locality property, not a feature. Two GPUs under one switch exchange data at full link rate without host memory involvement. The same two GPUs on opposite sockets may run at a sixth of that, or may not support peer-to-peer at all, in which case the driver silently falls back to staging through host DRAM — two transfers instead of one, plus a memory copy.

NUMA is not only a CPU concern. The inter-socket link exists to keep caches coherent. It is not a data path, and treating it as one is how people end up with a cluster that benchmarks at a third of expectation for no visible reason.

GPU-to-NIC affinity is the same rule wearing a different hat. If a GPU's outbound traffic has to climb to the root complex to reach a network card, you have added a slow hop to every byte that leaves the node. Well-built nodes pair each GPU with a NIC under a shared switch, and this pairing is exactly what makes the rail-based topologies of §9 possible.

What a lane actually delivers

The other half of PCIe is arithmetic. A lane is one differential pair in each direction, a link is some number of lanes striped together, and the headline transfer rate is not the rate you get.

A PCIe lane is one differential pair in each direction. Gen5 clocks it at 32 GT/s — 32 billion symbols per second. But not every symbol is payload. Through Gen5, PCIe uses 128b/130b encoding: every 130 bits on the wire carry 128 bits of data plus a 2-bit sync header. So one Gen5 lane delivers

Multiply by 16 lanes and you get 63 GB/s each way for a x16 slot. Gen6 changes the recipe — PAM4 signalling puts two bits per symbol so 64 GT/s comes from the same 32 GBd baud rate, and the 128b/130b scheme is replaced by fixed 256-byte FLITs with built-in forward error correction — landing at roughly 121 GB/s.

The practical consequence is that on any node where the NIC hangs off the same PCIe root complex as the GPU, the network's ceiling is PCIe's ceiling. A p5 instance pairs each pair of GPUs with enough EFA capacity that this is balanced, but on ad-hoc clusters it is extremely common to find a 400 Gb/s NIC (50 GB/s) sitting behind a Gen4 x16 link (31.5 GB/s), which quietly caps the network at 63% of its rating. Check lspci -vv before you trust a spec sheet.

NVLink

PCIe already connects every device in the machine. So the obvious question is: why build a second interconnect at all?

Look back at the ladder. A Gen5 ×16 link gives 63 GB/s. HBM gives 3,350 GB/s. If a GPU has to fetch an operand from a neighbour over PCIe, it does so at 1.9% of the rate it reads its own memory. Any strategy that splits a single layer's work across GPUs — where operands genuinely have to move every step — is dead on arrival at that ratio.

So NVLink exists to answer one specific need: make a small group of GPUs behave, as nearly as possible, like one large GPU. Three design choices follow from that goal.

Much more bandwidth

NVLink 4 on Hopper uses 18 links per GPU at 25 GB/s per direction each, giving 450 GB/s per direction per GPU — about seven times a Gen5 ×16 link. That moves the break-even arithmetic intensity from 15,700 down to 2,200, which is the difference between impossible and merely demanding.

A memory-semantic protocol

PCIe carries packets with substantial headers, designed for arbitrary peripherals. NVLink carries loads, stores and atomics — the operations a GPU already speaks. Less framing overhead, and a remote GPU's memory can be addressed rather than messaged.

Direct paths that skip the root complex

NVLink connects GPUs to each other without climbing the PCIe tree, so none of the root-complex and NUMA penalties from §4 apply. This is why the same pair of GPUs can be either fast or slow neighbours depending purely on which fabric carries the traffic.

Those are the claims. The figure below is the argument — the same transfer, raced down each path, with the numbers live.

The middle two lanes are the point. PCIe Gen5 gives 63 GB/s and the network gives 50 GB/s per GPU — a 1.26× difference. PCIe is network-class bandwidth that happens to live inside the chassis. It is entirely adequate for loading weights or shipping a batch, and hopeless as a substrate for work where operands move every layer. NVLink is what turns a chassis of eight separate accelerators into something that can be treated, for some purposes, as one.

The 9× number is the one to carry forward. It is the cliff between "inside a node" and "outside a node", and essentially every placement decision in distributed training is an attempt to keep the heaviest traffic on the fast side of it.

NVSwitch

NVLink gives us fast links. It does not yet give us a fast node, and the gap between those two things is worth deriving carefully, because it is the same argument that will reappear at cluster scale in §9.

The connectivity problem

Suppose you have N GPUs and you want any one of them to be able to talk to any other at full bandwidth. The obvious approach is to run a direct link between every pair. Count the links:

For N = 4 that is 6 links and 3 per GPU — easy. For N = 8 it is 28 links and 7 per GPU. For N = 16, 120 links and 15 per GPU. The link count grows quadratically, but the binding constraint is worse than that: each GPU has a fixed budget of links, set by die edge area and power. Hopper has 18. Once N−1 exceeds what you can physically attach, full direct connectivity is simply unavailable.

You then face an unappealing choice. Either connect only some pairs — a mesh — and accept that the unconnected pairs must take two hops through an intermediate GPU, at half bandwidth and with someone else's GPU doing the forwarding. Or split your fixed link budget across more neighbours, so every pair is connected but each connection is thinner.

This is the same problem telephone networks faced in the 1950s, and the answer is the same: stop trying to connect endpoints to each other, and connect them all to a switch instead. Every GPU spends its entire link budget on one connection — to the switch — and the switch provides any-to-any routing internally. Link count per GPU becomes 1 regardless of N, and any pair achieves full bandwidth in one hop.

The reason NVSwitch matters is not that it is faster than a mesh in the best case — a direct mesh link is as fast as a switch port. It is that it makes every case the best case. On a Volta mesh, a ring collective had to be laid out along physical links or it would traverse two hops and halve its bandwidth; NCCL contained substantial machinery to discover the topology and construct rings that respected it. With a full crossbar, any permutation of GPUs achieves full bandwidth, and the collective library's job collapses from "find a good embedding" to "pick any ring."

The current extreme of this idea is rack-scale NVLink: GB200 NVL72 puts 72 GPUs into one NVLink domain with a switch fabric between racks, which turns what used to be a network problem into an intra-node problem for models that fit inside 72 GPUs. That is a topology change with real consequences for parallelism strategy, and we will come back to it.

The reason NVSwitch matters is not that it beats a mesh in the best case — a direct mesh link is exactly as fast as a switch port. It is that it makes every case the best case. On a Volta mesh, a ring collective had to be laid out along physical links or it would take two hops and halve its bandwidth, and NCCL carried substantial machinery to discover the topology and build rings that respected it. With a full crossbar, any permutation of GPUs achieves full bandwidth, and the library's job collapses from "find a good embedding" to "pick any ring."

The current extreme of this idea is rack-scale NVLink: GB200 NVL72 places 72 GPUs in a single NVLink domain with a switch fabric between racks. That converts what used to be a network problem into an intra-node problem for any model that fits inside 72 GPUs — which is a topology change with real consequences, and it brings us to the distinction the next section is about.

Scale-up and scale-out

We now have enough to name the most consequential distinction in the whole subject. It is not a distinction between technologies so much as between two regimes, separated by a bandwidth cliff of roughly 9×.

Scale-up

Making one tightly coupled domain bigger. Within it, every GPU reaches every other at hundreds of gigabytes per second over NVLink and NVSwitch, in one hop, with latency around 2 µs. Historically this meant one chassis of 8 GPUs; with rack-scale NVLink it can now mean 72. The domain is bounded by what a switch fabric can physically reach.

Scale-out

Adding more domains. Between them you have InfiniBand, Ethernet with RoCE, or something EFA-like: tens of gigabytes per second per GPU, several switch hops, latency of 3–12 µs, and traffic that shares the fabric with everyone else's. This direction has essentially no ceiling — clusters run to tens of thousands of GPUs — but every byte costs about nine times more.

The reason this matters more than any individual specification is that the two regimes are not interchangeable. You cannot buy your way from one to the other; the cliff is structural. So the question stops being "how many GPUs do I have" and becomes "how are they grouped, and does my communication pattern respect the grouping?"

Everything downstream in this article is a consequence of that cliff.

• Collective algorithms are worth decomposing by level, because a flat algorithm treats the fast domain and the slow fabric identically and therefore wastes the fast one entirely (§13).
• Topology matters — rails, oversubscription, bisection bandwidth — because scale-out is where all the interesting failure modes live (§9, §15).
• And the entire question of which parallelism strategy to use reduces, in large part, to which strategies generate traffic that fits inside a scale-up domain and which do not.

That last point is the subject of the next blog, not this one. Here it is enough to have the vocabulary and the number: 9×, at the edge of the node.

NIC, RDMA and GPUDirect

Scale-out means the data has to leave the machine, which means it has to go through a network card. Start by writing down the path a byte takes under the obvious, traditional design:

Every arrow is real work. The GPU copies to a host buffer over PCIe; the kernel copies that into a socket buffer; the NIC reads it back over PCIe; and the whole thing runs in reverse at the far end. Which invites the obvious question: why is host memory involved at all? The CPU is not transforming the data. It is acting as a relay for bytes that were already sitting in one device and are destined for another.

Removing it happens in three steps, each solving a different problem.

DMA

Direct memory access: the device moves the bytes itself instead of the CPU issuing loads and stores. This removes the CPU from the copy, but the data still lands in host memory.

RDMA

Remote DMA extends that across the network. One machine's NIC writes directly into another machine's memory, with no involvement from the remote CPU at all — no system call, no interrupt, no kernel buffer. The remote side does not even have to be listening. This removes the CPU from the transfer.

GPUDirect RDMA

The last step: the NIC reads and writes GPU memory directly, over PCIe peer-to-peer. Host memory disappears from the path entirely. This removes host memory from the route.

Counting PCIe crossings makes the payoff concrete. The staged path crosses PCIe four times per direction; GPUDirect crosses it twice. Each removed crossing also removes a buffer, a synchronisation point, and a chunk of α.

InfiniBand versus EFA

Both are RDMA fabrics, but they make opposite bets about ordering, and that single difference explains most of their behavioural differences.

InfiniBand's Reliable Connection transport delivers packets in order. That is a lovely property for software, and it means a connection follows one path through the fabric, chosen by a deterministic hash of the packet headers. It also means that if two heavy flows hash to the same link, they collide and stay collided for the life of the flow.

AWS's Scalable Reliable Datagram transport takes the other side. SRD sprays packets across every available path and delivers them out of order, pushing reassembly up to the library. This trades ordering guarantees for near-immunity to hash collisions, and it makes tail latency dramatically more predictable at scale. The cost shows up in α: EFA's per-message latency is meaningfully higher than InfiniBand's, in the region of 10–15 µs versus 2–3 µs.

The practical reading for training workloads is that EFA punishes small messages and rewards large, well-pipelined transfers. Which is another way of saying: on EFA, keep your collectives above n_½, and prefer strategies whose inter-node traffic arrives in large contiguous chunks.

Network topology

Scale-out gives us more domains. It does not say how they are wired, and that turns out to matter enormously. A cluster is not a bag of nodes — it is a graph, and the shape of the graph decides which communication patterns are cheap and which are catastrophic.

The dominant design for GPU clusters is a folded Clos network, usually called a fat tree. Endpoints attach to leaf switches; leaf switches attach to spine switches; every leaf connects to every spine. The defining property of a non-blocking Clos is that the aggregate uplink capacity of each leaf equals its downlink capacity, so any permutation of endpoints can communicate at full rate simultaneously. Building this with radix-k switches gives you k³/4 endpoints in three levels — with 64-port switches, 65,536 GPUs.

Rails

The refinement that matters for GPU training is rail-optimised wiring. Instead of connecting all eight NICs of a node to the nearest leaf switch, you connect GPU j of every node to leaf switch j. Leaf switch 3 therefore sees GPU 3 of every node in the cluster and nothing else.

Why bother? Because the collectives that dominate training are symmetric across nodes. When 64 GPUs perform an AllReduce, the traffic pattern is not random: rank 3 on node A talks to rank 3 on node B. Under rail-optimised wiring that entire conversation lives inside one leaf switch — one hop, no spine, no contention with the other seven rails. The spine only carries traffic that genuinely needs to change rail.

Oversubscription and bisection

Bisection bandwidth is the aggregate capacity crossing the worst-case cut that splits the cluster in half. It is the hard ceiling on any all-to-all pattern, because in the worst case half the GPUs are sending to the other half. A fully non-blocking fat tree has bisection equal to half the endpoint bandwidth; real clusters are frequently oversubscribed at the leaf by 2:1 or 4:1 because non-blocking spine capacity is expensive and most workloads do not need it.

Most workloads. Mixture-of-experts training does need it, because expert routing produces genuine all-to-all traffic with no locality to exploit. If you are planning an MoE run on a cluster someone else specified, the oversubscription ratio is the first number to ask for.

Collective communication

A collective is an operation that every rank in a communicator participates in. It is not a library convenience; it is a different primitive from point-to-point messaging, because the implementation is free to choose any pattern of point-to-point transfers that produces the required result. That freedom is where all the performance is.

There are eight that matter, plus a barrier. Before we look at algorithms, look at semantics — what each one promises, independent of how it is achieved. In the figure, colour indicates which rank a piece of data originated on, and Σ indicates a reduction.

Two identities worth memorising

The eight are not independent. Two decompositions do almost all the work in real implementations:

The first is genuinely how every bandwidth-optimal AllReduce is built. Reduce the vector so that each rank owns the finished sum of one slice, then let everyone gather all the slices. Each half costs (p−1)/p · n bytes of traffic per rank, so the whole thing costs 2(p−1)/p · n. We will prove in a few sections that you cannot do better.

The second identity is true and useless: routing everything through one rank makes that rank's link the bottleneck and gives you a cost proportional to p·n instead of n. It is a good reminder that the semantics of a collective say nothing about its cost.

Deriving AllReduce

AllReduce is the one to derive properly. It is the most common collective in training, it is the one with the most interesting algorithms, and once you have derived it the others follow almost immediately.

The specification: every rank starts with a vector of n bytes, and every rank must finish holding the elementwise sum of all p vectors.

Start with the obvious algorithm

The way anyone would do it by hand: pick a rank, have everyone send their vector to it, let it add them up, then have it send the result back to everyone.

It is correct, and it is terrible. Count the traffic at rank 0: it receives (p−1)·n bytes and then sends (p−1)·n bytes, while every other rank moves only 2n. With p = 64 and n = 100 MB, rank 0 handles 12.6 GB while everyone else handles 200 MB and waits.

The cost is therefore

which grows linearly in p. Double the cluster and the collective takes twice as long. That is the opposite of what you want from adding hardware, and it happens for one identifiable reason: a single link is being asked to carry all the traffic while p−1 other links sit idle.

Everything that follows is an attack on that one flaw. There are two independent ways to attack it, and they lead to two different families of algorithm.

Attack one: use more links at once — the tree

Instead of everyone sending to one rank, pair the ranks up. In round one, odd ranks send to even ranks and each even rank adds what it received. Now half the ranks hold partial sums. Repeat. After ⌈log₂p⌉ rounds one rank holds the total, and reversing the process broadcasts it back.

Every round uses p/2 links simultaneously rather than one, and the round count is logarithmic rather than linear:

At p = 64 that is 12 rounds against the naive 126 — a tenfold improvement, and it gets better as p grows. But look closely at the β term: every round still moves the entire vector. We fixed the latency scaling and left the bandwidth scaling alone.

Attack two: send less — the ring

The second flaw in the naive algorithm is that each rank sends its whole vector to one place. But a sum can be computed piecewise. If we cut the vector into p chunks, then chunk i can be summed on rank i — different chunks reducing on different ranks, in parallel, with no rank ever holding more than 1/p of the data.

That gives a two-phase structure, and both phases are collectives you already met in §10:

ReduceScatter leaves each rank holding one fully reduced chunk. AllGather then circulates those finished chunks so everyone ends up with all of them. This identity is the single most useful structural fact about collectives, and we will use it repeatedly.

Before deriving the cost, look at the three broadcast strategies side by side, since the same three ideas — flat, tree, pipelined ring — are exactly what we are choosing between.

Reduce is broadcast run backwards with arithmetic at the merge points, and every one of these three schemes has an exact mirror image. The costs are identical with a γn term added per level. That symmetry is worth internalising, because it means anything you learn about one applies to the other.

The ring, step by step

The ring AllReduce is the single most important algorithm in distributed deep learning, and it is worth deriving rather than quoting.

Setup: p ranks in a cycle, each holding a gradient vector of n bytes. Split each rank's vector into p chunks of n/p bytes each. Index them 0 through p−1, the same way on every rank.

Phase one: reduce-scatter

Run p−1 rounds. In round s, rank i sends chunk (i − s) mod p to rank i+1, which adds it into its own copy of that chunk. Follow one chunk: chunk 0 starts on rank 0, moves to rank 1 in round 0 picking up rank 1's contribution, to rank 2 in round 1, and so on. After p−1 rounds it has visited every rank and accumulated every contribution — and it now sits on rank p−1.

Every rank is doing this simultaneously for a different chunk, so at the end of phase one, rank i owns the fully-reduced chunk (i+1) mod p and nothing else is finished. That is exactly a ReduceScatter. Each rank sent p−1 chunks of n/p bytes:

Phase two: all-gather

Now run p−1 more rounds of the same shape, except the receiver overwrites instead of adding, because the arriving chunk is already final. In round s, rank i sends chunk (i − s + 1) mod p to rank i+1. After p−1 rounds every rank has every finished chunk.

Add them up and apply the α–β model, counting 2(p−1) rounds each of which is one message of n/p bytes:

Two properties are worth pausing on. First, the bandwidth term is independent of p in the limit: as p → ∞, 2(p−1)/p → 2, so a ring AllReduce across ten thousand GPUs moves the same number of bytes per GPU as one across ten. Second, every link is busy in every round. There is no idle capacity anywhere in the schedule. Those two facts together are why the ring survived.

Step through it below. Each cell shows which ranks' contributions have been folded into that chunk; green means the chunk is complete.

Is the ring as good as it gets?

Is 2(p−1)/p · n the best possible? Yes, and the proof is short enough to do here. Assume each rank has one full-duplex link and can send and receive simultaneously — the standard model.

AllGather first

In an AllGather, each rank starts with n/p bytes and must end with all n bytes. The (p−1)/p of the result it does not already hold cannot be manufactured locally; it must arrive over the wire. So every rank must receive at least

and by symmetry (the union of all ranks' data is the result, and every rank needs it) every rank must send at least as much. A ring AllGather transmits exactly (p−1) chunks of n/p, which is (p−1)/p · n. Optimal, with nothing left on the table.

ReduceScatter by duality

ReduceScatter is the transpose of AllGather: reverse every message and replace concatenation with addition. Any ReduceScatter schedule run backwards is a valid AllGather schedule with the same volume, so the same bound applies. Rank i's output slice depends on all p ranks' contributions to that slice, and p−1 of those live elsewhere, so at least (p−1)/p · n bytes must be received per rank.

Composition

An AllReduce implies a ReduceScatter (just throw away the slices you do not want) and implies an AllGather (start from a state where each rank owns one finished slice). Both are subproblems, so:

The ring meets this exactly. Recursive halving-doubling also meets it exactly. Any algorithm claiming to beat it is either changing the problem — compressing, quantising, sparsifying, or reducing inside the network so that a switch does work a rank would otherwise do — or measuring wrong.

So which one should you use?

We now have a latency-optimal family (trees) and a bandwidth-optimal family (rings), and the honest answer is that neither wins outright — it depends on n. The comparison below is live; drag the sliders and watch the crossover move.

Play with it for a minute and the structure becomes obvious. With α = 12 µs (EFA-like) and 45 GB/s, at p = 64 the tree wins below roughly 100 KiB and the ring wins above it. Push α down to 2 µs (InfiniBand-like) and the crossover moves left. Push p up to 1024 and the ring's 2(p−1)α term — over 24 milliseconds of pure latency at α = 12 µs — makes it hopeless for anything small, which is exactly why NCCL ships a tree algorithm and switches between them.

AllToAll

AllToAll is the collective with no shortcuts. Every rank sends a distinct message to every other rank — it is a distributed transpose — and unlike a reduction there is nothing to combine on the way, so no clever tree can reduce the volume.

Per-rank volume is (p−1)/p · n, which looks cheap next to AllReduce's 2(p−1)/p · n. The problem is not volume, it is structure. An AllReduce has locality you can exploit: reduce within the node first, then across. An AllToAll has none — every pair genuinely must exchange data, so the traffic is spread uniformly across the fabric and the limiting resource is bisection bandwidth, not per-link bandwidth. On an oversubscribed fat tree, AllToAll is the pattern that finds the oversubscription.

The mixture-of-experts case is worth flagging now because it breaks an assumption baked into every cost model in this article: that message sizes are known and equal. In expert routing they are neither. The number of tokens a rank sends to a given expert depends on the router's output, which depends on the data. Load imbalance means some ranks wait; capacity factors and auxiliary load-balancing losses exist precisely to bound that imbalance, and they are a communication optimisation dressed up as a modelling choice.

We will not go further into expert routing here — the point for now is only that All-to-All exists, that it is the least forgiving collective, and that one important class of model leans on it heavily. The next blog picks that thread up.

Hierarchical collectives

Every algorithm in §11 assumed one flat network with a single β. Section 7 told us that is false: real clusters have at least two regimes differing by roughly 9×. A collective that ignores the distinction leaves most of the machine unused, and it is worth seeing exactly how.

Consider a flat ring AllReduce across 16 GPUs on 4 nodes. Order the ring so that consecutive GPUs are neighbours where possible, and you still cross a node boundary four times. Now here is the problem: a ring runs at the speed of its slowest link. Every round, every link must carry one chunk, so the twelve NVLink hops sit idle at 11% utilisation waiting for the four network hops. The entire collective proceeds at network speed even though three quarters of it is on NVLink.

The fix is to decompose. Reduce inside each node over NVLink, exchange only the reduced slices over the network, then broadcast back inside the node.

Written out, the hierarchical cost for G GPUs per node and N nodes is

against the flat ring's 2(P−1)/P·n·β_net with P = N·G. The network term shrinks by a factor of G·(P−1)/P·N/(N−1), which for realistic sizes is just about exactly G — the number of GPUs in the fast domain. On an 8-GPU node that is an 8× reduction in fabric traffic before you account for the fact that the NVLink phases also run 9× faster.

NCCL does this automatically when it can see the topology, which is one reason its tree algorithm has a two-level structure: an intra-node phase and an inter-node phase with one representative GPU per node per rail. It is also, structurally, exactly what HSDP does with its sharding and replication dimensions.

An aside: reduction inside the switch

There is one more move available: do the arithmetic in the network. InfiniBand switches with SHARP, and NVSwitch 3 with NVLink SHARP (marketed as NVLS), can perform the reduction on the aggregation tree itself rather than shuttling partial sums between endpoints.

It is worth being precise about what this buys, because the marketing tends to overstate it. Under in-network reduction each rank sends its full vector up once and receives the result down once, so per-rank volume is 2n. The ring's volume is 2(p−1)/p·n, which for p = 64 is 1.97n. So the bandwidth advantage is essentially zero — in fact the ring moves marginally fewer bytes.

The real wins are elsewhere, and they are substantial:

• Latency. The 2(p−1)·α term collapses to something like 2·α plus switch processing. At p = 512 and α = 3 µs that is 3 milliseconds of pure latency removed from every step.
• GPU cycles. A ring AllReduce burns SMs on copy and add kernels for its entire duration, competing with the compute you are trying to overlap it with. In-network reduction gives those back.
• Fabric traffic. Volume per rank is unchanged, but the traffic no longer traverses p−1 hops of the tree — the switch consumes p inputs and emits one, so aggregate bytes on the wire drop sharply even though per-endpoint bytes do not.

The constraints are real: the reduction operator must be one the switch implements, the datatype must be supported, and the aggregation tree must be configured by the fabric manager. It is not something you get by accident, and on clusters without it, none of this applies.

Overlap

Every cost in this article so far has been treated as time you spend. That assumption is wrong, and correcting it is the single largest lever available.

Communication and computation use different hardware. A collective moving bytes over NVLink or the network does not need the tensor cores; a matrix multiply does not need the NIC. There is no physical reason they cannot happen at the same time. And if a communication finishes while the GPU is still busy with arithmetic it was going to do anyway, then that communication cost nothing — not less, nothing. The α–β model still describes it, but the answer no longer appears in your step time.

Three things have to be true for that picture to be achievable, and each corresponds to something real in the software.

The operations must be independent

You can only overlap a collective with compute that does not depend on its result. In a backward pass this is naturally satisfied: gradients become available in reverse layer order, so layer 31's gradient can be reducing while layer 30 is still computing. The dependency structure of backpropagation is what makes deep-learning communication hideable in the first place.

The hardware must run them concurrently

This is what CUDA streams provide. A collective issued on a separate stream from the compute kernels executes asynchronously, and the copy engines and NIC operate without blocking the tensor cores. Nothing overlaps by accident — if everything is issued on the default stream, everything serialises.

The work must be divisible

Overlap only exists if there is something to interleave. One monolithic collective at the end of the step has nothing to hide behind, which is why gradients are accumulated into buckets and reduced in batches as they become ready.

That last point has a tension in it that the next figure exists to explore. More, smaller buckets start communicating earlier and overlap better — but each one pays α again, and from §3 we know α does not care how small the message is. Fewer, larger buckets amortise α but delay the first launch and leave a longer tail. There is an optimum, and it is not where most people guess.

Bucketing, and the cost of getting it wrong

The naive approach launches one AllReduce per parameter tensor. A 7B model has thousands of tensors, many of them small: layer-norm gains, biases, embeddings split across ranks. Each collective costs 2(p−1)α in latency regardless of size, so at p = 64 and α = 12 µs, every message costs 1.5 ms before a single byte moves. A thousand tensors is a second and a half of pure latency.

So gradients are accumulated into buckets and reduced in batches. This is the same trade-off as the pipelined broadcast in §11: more, smaller messages give better overlap but pay more α; fewer, larger messages amortise α but delay the first launch and leave a longer tail. There is an optimum, and it moves with p.

Three things the timeline makes obvious that the algebra does not.

The tail is structural. The last bucket contains the gradients of the first layers, which do not exist until the backward pass is essentially over. No amount of tuning removes it; the best you can do is make it small. This is why the final bucket is often deliberately sized smaller than the rest.

The optimum shifts with p. Drag the rank count from 64 to 1024 and watch the α term dominate: the optimal bucket grows, because each message now costs 2·1023·α ≈ 25 ms in latency alone and you can afford far fewer of them. PyTorch's DDP default of 25 MB is tuned for modest scale; at four-digit rank counts it is much too small.

Bandwidth changes the shape, not the location. Halving bandwidth scales the whole curve up but barely moves the minimum, because the optimum is set by the ratio of α to β, not by either alone — exactly as in the pipelined-broadcast derivation where k* = √((p−2)nβ/α).

Congestion and contention

Every bandwidth figure quoted so far is a capacity. What you get is capacity divided by however many flows happen to be sharing the link, and in a datacentre fabric that divisor is decided by a hash function you do not control.

ECMP and the birthday problem

A leaf switch with several uplinks to the spine layer must choose one for each packet. Choosing per-packet would reorder the flow, which a reliable-connection transport cannot tolerate, so the standard answer is ECMP: hash the five-tuple — source and destination IP, source and destination port, protocol — and send the whole flow down whichever uplink the hash selects. Order is preserved because every packet of a flow follows one path.

The cost is that path selection is random and uncoordinated. Two flows can hash to the same uplink while a neighbouring uplink sits idle, and no mechanism notices. This is the balls-in-bins problem: throw k flows into k uplinks and the expected occupancy of the busiest is not 1 but roughly ln k / ln ln k. At k = 64 that is about 2.9 — the unluckiest link carries nearly three times its share, and since a collective runs at the speed of its slowest participant, that link sets the pace for everyone.

Two answers, and what each costs

Spread the packets. AWS's SRD, used by EFA, abandons the idea that a flow owns a path. It sprays packets of a single message across every available route and reassembles at the receiver. Load is even by construction and hash collisions cannot happen. The cost is that the transport must handle arbitrary reordering, which is why EFA exposes an unordered delivery model and why its α is roughly four times InfiniBand's — the reordering and retransmission logic lives above the wire. NVIDIA's newer InfiniBand generations offer adaptive routing that achieves something similar by moving flows off congested paths, and the RoCE world has been converging on spraying too.

Never leave the rail. If GPU j on every node communicates only with GPU j on other nodes, the traffic is leaf-local or single-path by construction and there is nothing to collide with. This costs nothing at all, which is why rail-aligned collectives are so much more predictable than their raw bandwidth suggests, and why NCCL works hard to keep inter-node phases on-rail.

Incast

The other failure mode is not a routing problem but a buffering one. When many ranks send to one — the gather step of a naive AllGather, or any AllToAll where the schedule accidentally synchronises — the last-hop switch port is oversubscribed by the fan-in ratio, and switch buffers are shallow, measured in tens of megabytes shared across all ports. At 400 Gb/s a port drains a megabyte in 20 microseconds, so a 32-way incast fills the buffer faster than any end-to-end congestion signal can react.

Lossless fabrics respond with backpressure: ECN marks to slow the senders, and if that is too slow, PFC pause frames to stop the upstream port outright. PFC works, but it pauses everything on that priority class, including traffic that was not part of the problem, and pauses propagate upstream. This is how a single hot receiver turns into a fabric-wide stall, and in the worst case a deadlock. It is the reason schedules matter: a ring AllReduce, where every rank sends to exactly one peer at a time, is nearly immune to incast, while a flat gather is a machine for producing it.

What we still have not asked

Pulling the thread back through: we started by noticing that reaching another GPU is the same kind of event as reaching your own memory, only further away. Everything after that was working out how much further, and what to do about it.

Message time has six parts, and only one of them scales with size

Which is why cost is affine, why α and β are the right two parameters, and why n_½ = α/β cleanly separates the two regimes you can be in.

PCIe is a tree, and distance in that tree is cost

Two devices under one switch are fast neighbours. The same two devices across sockets may be six times slower or refuse peer-to-peer entirely.

NVLink exists to make a small group of GPUs behave like one big one

And NVSwitch exists because direct wiring needs N−1 links per GPU, which runs out long before N does.

The node boundary is a 9× cliff

Scale-up and scale-out are different regimes, not different sizes, and no amount of money moves the boundary.

Collectives are specifications, and the algorithm is a free choice

The naive AllReduce is linear in p because one link carries everything. Trees fix the latency term, rings fix the bandwidth term, and rings provably cannot be beaten on bytes moved.

Flat algorithms waste hierarchical machines

A ring advances at the pace of its slowest link, so decomposing by level cuts fabric traffic by roughly the size of the fast domain.

Communication you can hide is nearly free, and the tail is what remains

Which turns most tuning into a question about bucket sizes and dependency structure rather than about bandwidth.

Where this goes next

Notice what this article carefully did not do. It described collectives in the abstract — as operations on some vector of n bytes — and never asked where that vector comes from. Every cost model above takes n and p as given.

But n and p are not given. They are consequences of how you chose to split a model across devices, and that choice is the actual design space. The mapping looks roughly like this, and every row of it deserves a derivation this article did not give:

So the honest state of things is this. We now understand compute, memory and communication as machinery. What we have not asked is what an ML training job actually creates: which tensors exist, how large they are, how long they live, and why any of them need to move at all. Without that, the cost models above are a calculator with no inputs.

That is Blog 04: Inside a Training Step.

The nuances, in one place

Seventeen claims in this article contradict something that is widely repeated. They are marked inline where they arise; this is the index, so a belief can be checked without rereading the argument around it. Each entry states the correction rather than the common version — follow the link for the reasoning.

1. 01Arithmetic intensity is per level, not per chip — 295 is the HBM number; the fabric's is nearer 19,800.
2. 02Latency is a software property, not a distance — the wire is nanoseconds; α is microseconds, and the gap is the stack.
3. 03PCIe headline rates are pre-encoding, and the encoding changed — Gen5 ×16 is 63.0 GB/s, and Gen6 re-does the framing entirely.
4. 04The NVLink-to-network gap is 9×, not 18× — 18× compares bidirectional against unidirectional.
5. 05An exact factor of two is almost never a bug — check five conventions before looking for a fault.
6. 06EFA's higher α is a design choice, not weaker silicon — you are buying immunity to hash collisions with latency.
7. 07There are two optimality floors, not one — ring hits the bandwidth bound and misses the latency bound.
8. 08A plateau at half line rate may be the protocol — LL is 50% efficient by construction, and on purpose.
9. 09The ring-versus-tree crossover is moot twice over — Rabenseifner wins both terms; hierarchy outranks both.
10. 10AllToAll moves half what AllReduce moves, and is still harder — the binding constraint is bisection, not volume.
11. 11MoE breaks the equal-message-size assumption — cost follows the worst imbalance, not the mean.
12. 12Hierarchy buys you G, not the bandwidth ratio — traffic falls by the fast-domain size; link speed is a separate factor.
13. 13In-network reduction is not a bandwidth optimisation — 2n versus 1.97n; sell it on latency and SM occupancy.
14. 14The bucket optimum tracks p, not bandwidth — 25 MB is far too small at four-digit rank counts.
15. 15Hidden communication still costs a few percent — overlapped means 3% instead of 30%, not zero.
16. 16ECMP collisions are the normal case — the busiest of k links carries about 2.9× its share at k = 64.
17. 17PFC pauses traffic unrelated to the congestion — which is how one hot receiver stalls a region of the fabric.

If there is a pattern across the seventeen, most come from one of three habits: a unit or direction convention left unstated (03, 04, 05), a quantity that is optimal in one term being described as optimal outright (07, 09, 13), or a cost model applied outside the assumptions it was derived under (06, 11, 15). All three are cheap to guard against once you know to look.

References

Grouped by what each source is useful for rather than by publication date. Links point to the primary papers, official documentation, and reference implementations behind the discussion.