How to Find What Is Slowing Your Model

Every optimisation is a hypothesis

Someone tells you the training run is slow. You have a list of things that make training runs faster: bigger batches, fused kernels, mixed precision, gradient accumulation, a better data loader, torch.compile, CUDA graphs, a faster interconnect. Every one of those is a real technique that has produced real speedups on real workloads. And on any particular run, most of them will do nothing at all.

They do nothing because a technique only helps if it relieves the resource that the machine is currently waiting on. Fusing elementwise kernels is a bandwidth optimisation; if you are waiting on the network it changes nothing. Overlapping communication is a scheduling optimisation; if your GPUs are idle because a Python worker is decoding JPEGs, overlapping the all-reduce with a GPU that is already idle is free and worthless. A speedup is not a property of a technique. It is a property of the match between a technique and a bottleneck.

So the sentence to hold onto for the rest of this article is:

Every optimisation is a hypothesis about the bottleneck. Profiling exists to test that hypothesis before you spend a week on it.

The seven articles before this one built a model of the machine: what work a layer implies, what tensors it materialises, where those tensors live, how many bytes move across which boundary, and who has to talk to whom when the model no longer fits on one device. This article closes the loop. It is about walking up to a running system that is slower than you expected and finding out, from evidence rather than folklore, which of those costs is the one you are actually paying.

The method has three moves, and they are the same three moves whether the workload is a 7B vision-language model on 64 H100s or a single-GPU inference server. Look at time: build a timeline and find where the scarce resource is not doing useful work. Name the resource: measure enough to say which of eleven things is saturated. Predict, then verify: state what the fix should buy you in units of time, apply it, and check whether the number moved. If it did not, your hypothesis was wrong, and that is information — as long as you were measuring.

Four kinds of callout run through this article. Profiling is a field where the confident wrong answer is the expensive one, so the callouts carry most of the warnings.

The timeline is the primitive

Almost every performance question is really a question about time, and time in a modern accelerator system is not one line but several. There is a CPU thread running Python and enqueuing work. There is a copy engine moving bytes over PCIe or NVLink. There are one or more compute streams executing kernels. There is a NIC moving gradients or activations between hosts. There is a storage path pulling shards off a network filesystem. These run concurrently, they depend on each other, and the thing you call "step time" is the shape their overlap happens to take.

So before any counter, any roofline, any tool: draw the lanes. The first diagnostic question in all of systems performance is not "how fast is my GPU" but:

Is the scarce resource doing useful work continuously, and if not, which lane is it waiting on?

The figure below is one training step of a mid-sized vision-language model, drawn as five lanes. It is a deliberately unhealthy step, because healthy steps teach you nothing. Step it through: watch the playhead sweep, then look at what the GPU lane is doing while the other lanes are busy.

Three things in that figure are worth naming, because they generalise to every workload you will ever profile.

First, the gaps are the measurement. The step took 120 ms and the GPU did useful work for 64 of them. Nothing about the kernels was slow. If you had spent the week making the forward pass 20% faster you would have bought 10 ms out of 120, an 8% speedup, while a 22 ms hole sat untouched. The distribution of idle time is the priority list.

Second, the cause of a gap is never inside the gap. A hole in the GPU lane contains, by definition, no information about itself. You find the cause by looking at what the other lanes were doing during it, and then at what those lanes were waiting on. Here the GPU waited on a copy, the copy waited on a pinned buffer, and the buffer waited on a worker process that was still decoding. Three lanes up, the real problem is a CPU one.

Third, dependency, not duration, decides what matters. The all-reduce lane is busy for 44 ms but only 16 of those cost you anything, because 28 ms of it sits underneath backward work that had to happen anyway. Comparing "time spent in NCCL" against "time spent in GEMMs" is meaningless. The only communication number worth reporting is exposed communication: the part that no computation was available to hide.

nsys profile -t cuda,nvtx,osrt,cudnn,cublas \
     --capture-range=cudaProfilerApi \
     -o step_trace  python train.py

# wrap the steps you care about, or you will capture the whole warmup
with torch.profiler.profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    schedule=torch.profiler.schedule(wait=5, warmup=3, active=6),
    on_trace_ready=torch.profiler.tensorboard_trace_handler("./tb"),
    record_shapes=True, profile_memory=True, with_stack=True): ...

Eleven ways to be slow

"The model is slow" is not a diagnosis, and neither is "we are GPU bound". To make progress you need a vocabulary with enough resolution that each entry implies a different fix. Here is the full list. Everything in the rest of this article is machinery for deciding which row you are in.

What GPU utilisation actually counts

The first number everyone reaches for is the one nvidia-smi prints, and it is the most misleading number in the stack. It is not a measure of how much of the GPU you are using. It is the fraction of sampled time intervals during which at least one kernel was resident on the device. One thread block, occupying one of 132 streaming multiprocessors, doing an elementwise copy of four thousand floats, will report the same 100% as a perfectly tuned GEMM saturating every tensor core in the machine.

This is not a bug; it is what the counter was designed for, back when a GPU was a graphics device and "is anyone using this card" was the question. But it means the number can only ever falsify: 40% utilisation proves you have a gap problem, while 100% proves nothing at all. The figure below runs three completely different kernels through the same instrument.

The three kernels differ in every way that matters and agree on the only number most dashboards show. Kernel A is latency bound: it occupies eight SMs for nine microseconds, and the interesting fact about it is not its duration but that the model launches nine hundred like it per step. Kernel B is bandwidth bound: every SM is busy, most warps are resident, and almost all of them are stalled waiting on DRAM. Kernel C is compute bound with a third of the occupancy of B, which is fine, because each of its warps is issuing tensor-core instructions that do 512 multiply-accumulates apiece.

What we want to measure, before any tool

Tools change every eighteen months. What you want out of them does not. If you derive the measurement list from the machine model rather than from a tool's menu, you will know what to look for in whatever the next profiler is called.

The machine has a small number of resources that can be the scarce one, and each has exactly two properties worth knowing: how much of it you used and how fast it could have gone. Every diagnosis is a ratio between those two.

Notice the last row. It needs no special tooling, it costs nothing, and it is the single most informative number you can compute early: add up every kernel's duration in a step and divide by the step's wall time. If that is 0.55, no kernel optimisation in the world can buy you more than a 1.8× speedup, and you should go look for the gaps. If it is 0.95, gaps are not your problem and you can start reading counters.

Three magnifications, three tools

Profilers are usually presented as a list of products. It is more useful to think of them as three magnifications of one microscope. You start at the scale of the training step, and you only descend when the level above has told you where to point.

Scale 1 — the framework profiler

PyTorch's profiler knows things no external tool does: which Python operator issued a kernel, what the input shapes were, which module in your nn.Module tree it belongs to, and how much memory the allocator handed out during it. That provenance is why it is the right first stop. Sort by self CUDA time, look at the top ten rows, and ask whether the ranking matches your mental model of the network. If a layer you thought was free is in the top three, you have learned something before opening a single counter.

Its limitation is that it is a framework's view of the world. It sums operator durations well and reasons about the space between them poorly: copy engines, other processes, NCCL internals and CPU-side stalls that happen outside an operator are all either invisible or approximate.

Scale 2 — Nsight Systems

This is the tool for everything that is not a kernel. It samples the OS scheduler, traces CUDA API calls separately from kernel execution, shows the copy engines as their own lanes, unrolls NCCL, and — critically — draws all of it on one shared clock across processes. The step timeline in Figure 01 is exactly what it produces. Use it whenever the question is "why is there a hole", "who is waiting for whom", or "is this overlapping like I think it is".

The habit worth building: annotate your code with NVTX ranges that match how you think about the model, so the trace reads as vit_forward → connector → llm_forward → loss → backward → optimizer rather than as forty thousand anonymous kernels.

Scale 3 — Nsight Compute

Only descend here when scales 1 and 2 have already told you that a specific kernel is worth a specific fraction of the step. Nsight Compute replays a single kernel many times with different counters enabled, which is why it is slow and why it is definitive. It answers precisely one class of question: given that this kernel is running, what inside the SM is preventing it from going faster. Warp stall reasons, memory-hierarchy throughput at every level, tensor-core pipe activity, and a per-kernel roofline all come from here.

Three numbers into a verdict

Earlier in this series, Why Fast GPUs Still Wait for Memory derived the roofline from first principles. Here we only need its operational form, which is a procedure rather than a picture. Measure three things about a kernel or a phase: the useful FLOPs it performs, the bytes it moves at the DRAM boundary, and how long it took. From those, arithmetic intensity is FLOPs ÷ bytes, achieved performance is FLOPs ÷ time, and the ceiling for that intensity is the smaller of the machine's compute peak and its bandwidth times that intensity.

Then read off which of three situations you are in. On the sloped roof: you are bandwidth bound, and the only thing that helps is moving fewer bytes. On the flat roof: you are compute bound, and the only thing that helps is fewer or cheaper FLOPs. Well below both roofs: you are neither, which is the most common and most misdiagnosed case, and it means the kernel is limited by latency, occupancy, launch overhead or something else that stops the machine from ever reaching a throughput limit at all.

That third case matters because it is where most unoptimised ML code actually lives, and because both instincts it triggers — "reduce FLOPs" and "reduce bytes" — are wrong there. Watch a real optimisation walk across the plot.

The diagnosis tree

Everything so far collapses into one decision procedure. It starts with the cheapest measurement in the stack and only asks for expensive ones when the cheap one has narrowed the field. Walk it below; each leaf names the evidence that confirms it, because a verdict you cannot confirm is a guess with better vocabulary.

The input pipeline

Input bottlenecks are the most common cause of a disappointing training run and the least glamorous to fix, which is roughly why they survive so long. The structure is always the same: a producer, a bounded queue, and a consumer that is far more expensive per unit than the producer and therefore far more expensive to starve.

The producer is a pool of worker processes doing decode, augmentation, tokenisation and collation on CPU cores. The queue is the prefetch buffer — num_workers × prefetch_factor batches deep. The consumer is the GPU. Since queue depth is bounded, the steady-state rule is brutally simple: if the producer's average time per batch exceeds the consumer's, the queue empties and stays empty, and every step pays the difference. Depth buys you tolerance for jitter, never for a throughput deficit.

Configuration A loses 16 ms per 96, or 17% of the run, and it loses it forever — the queue never recovers, because recovery requires the producer to be faster than the consumer for a while and it never is. Configuration B doubles the workers and the queue saturates at its cap, which is what a healthy pipeline looks like: depth pinned near the top, with the occasional dip when a slow shard arrives. Configuration C moves decode to the GPU and makes the copy asynchronous, which does not make training faster at all — the GPU was already fully fed — but it frees eight CPU cores you can now spend elsewhere.

Distributed training

Once there is more than one GPU, a new class of bottleneck appears that has no analogue on a single device: waiting for someone else. It is also the class most often misdiagnosed, because the profiler faithfully reports where the time was spent — inside a collective — and the cause is usually somewhere else entirely.

Start from the dependency structure. Gradients become available progressively during the backward pass, in reverse layer order. A collective on the last layer's gradients can therefore start while the rest of the backward pass is still running, which is what gradient bucketing exists to exploit. The consequence is that only the tail of communication is ever exposed — unless something breaks the overlap.

Scenario three is the one worth internalising. Every rank's profile shows a long, slow all-reduce. The bus bandwidth during the transfer is at link peak. The network is innocent: three ranks reached the collective and then sat inside it, doing nothing, until the fourth arrived. If you respond by tuning NCCL you will change nothing, because the eighteen milliseconds you are chasing were spent on rank 2's forward pass — perhaps because its shard of the batch had more images, or longer sequences, or one unlucky sample with a very high resolution.

Scenario four is the measurement that separates them, and it is worth building into your harness permanently: put an explicit barrier immediately before the collective and time both halves. Time in the barrier is arrival skew, which is a load-balancing problem. Time in the collective after the barrier is transfer, which is a network problem. Without that split, the two are indistinguishable in every profile you will ever take.

Memory diagnosis

Memory is the bottleneck that does not slow you down; it stops you, or it forces a configuration that slows you down elsewhere. A batch size chosen because 33 would OOM is a memory bottleneck wearing a throughput costume, and so is every activation checkpoint you did not want.

Three numbers, and the difference between them is the whole subject. Allocated is what your tensors currently hold. Reserved is what the caching allocator has taken from the driver and will not give back — it keeps freed blocks in its own pools so the next allocation does not need an expensive, synchronising cudaMalloc. Peak is the high-water mark of allocated within a window, and it is the number that decides whether you fit. Reserved minus allocated is cached-but-idle memory, which is normally healthy and occasionally fatal.

The failure mode in that figure — an out-of-memory error while gigabytes sit free — confuses people the first time and is entirely mundane once you see the shape. The allocator hands out blocks carved from larger segments. Freeing a tensor returns its block to the pool, not to the driver. If your step allocates a different mix of shapes each iteration, those pools fill with fragments that are individually too small for the next large request. Variable sequence lengths, variable image resolutions and dynamic batch composition are the usual culprits, which is precisely why multimodal training hits this more than plain language modelling.

Inference diagnosis

Serving has its own vocabulary because it has its own objective. Training optimises one number, throughput. Serving optimises three that trade against each other: time to first token, time per output token, and total throughput. A configuration that maximises the third invariably damages the first, which is why "our GPUs are 95% busy" is a much weaker statement in serving than it sounds.

The structural reason is that a decoder does two very different jobs. Prefill processes the whole prompt in parallel: a big, compute-bound GEMM that saturates the tensor cores. Decode produces one token per request per iteration: a small, bandwidth-bound pass whose cost is dominated by reading the weights and the KV cache out of HBM. They compete for the same device, and how you schedule that competition is your latency profile.

Map the symptom to the mechanism before touching a knob:

Overlap, and what a step really costs

The resource model is now nearly complete, and the last piece is the one that makes all the arithmetic non-obvious. Compute, communication and data loading use different hardware. If they were perfectly independent, a step would cost the maximum of the three rather than the sum. If they were perfectly dependent, it would cost the sum. Real systems sit somewhere between, and where exactly they sit is decided by dependencies you can usually change.

So the useful decomposition of a step is not "how much time did each resource use" but:

step = critical-path compute + the part of everything else that had nothing to hide under.

That second term is the only one worth optimising, and it explains several results that otherwise look paradoxical. Making the backward pass faster can make a step less efficient, because there is now less computation available to hide the same collectives under. Adding gradient accumulation reduces communication per sample but also reduces the compute available to overlap it. Prefetching further ahead helps nothing if the producer is fundamentally slower than the consumer. Play with the arithmetic below until those three sentences feel obvious.

Case study: whack-a-mole

Here is the whole method applied to one run, and the point of it is the shape of the sequence rather than any individual fix. Each intervention is chosen because a measurement identified it, each is verified by re-measuring, and each one promotes a different resource to the top of the list.

The run went from 380 ms to 166 ms per step, a 2.29× speedup, and not one of the four moves was chosen from a list of best practices. Each was chosen because a measurement pointed at it. Note also what did not happen: nobody wrote a custom kernel, nobody changed the model, and the total time spent was days rather than weeks — because the expensive tools were only ever pointed at things the cheap tools had already implicated.

The last stage is the one people skip. After three fixes, the step is 90% dense GEMM running at a healthy fraction of the machine's peak. There is still headroom, in the sense that 61% is not 100%. But the remaining gains require changing the algorithm, the numerics or the parallelism strategy — and each of those carries correctness risk that the previous four fixes did not. Knowing that you have arrived at that boundary is as much a part of the discipline as finding the first bottleneck.

Ten questions

This is the last article in the series, so it should end with the thing that survives after the specific tools have been replaced. Every question the previous seven articles asked can be recovered from ten, asked in this order, of any machine learning system you are handed:

1. What computation must happen? Not what the code does — what the model definition implies. FLOPs by layer, from shapes.
2. What tensors exist? Parameters, gradients, optimiser state, activations, caches. Their sizes are arithmetic, not mystery.
3. Where do they live? Registers, shared memory, L2, HBM, host RAM, another device, disk. Each has a different price.
4. How many bytes move? Per operator and per step, at the boundary you name.
5. Across which boundary? The same gigabyte costs microseconds over NVLink and milliseconds over Ethernet.
6. Who communicates with whom? Which ranks, which collective, how many times per step, and at what point in the dependency graph.
7. What depends on what? The critical path is the only path whose length is your step time.
8. What can overlap? Anything with no dependency between it and something else — which is more than your current code allows.
9. What resource is saturated? Measured against its ceiling, not against your expectations.
10. If we remove that bottleneck, what becomes the next one? Ask before you start, so you know what the fix is worth.

Question ten deserves the last word, because it is the one that turns a technique into a discipline. Amdahl's arithmetic is unforgiving: a component that is 30% of your step time can never buy you more than 1.43×, no matter how brilliantly you eliminate it. Knowing the answer to question ten before you begin tells you whether the week is worth spending, and knowing it afterwards tells you when to stop.

Performance in machine learning systems is not about maximising GPU utilisation in isolation. It is about orchestrating compute, memory, communication, I/O and synchronisation so that the scarce resource stays useful while everything else stays out of its way — and then, when it finally is, having the discipline to stop.

References

Grouped by topic. Every entry links to its primary source.