Animated Deepdives
The Backward Pass
First-principles, illustrated deep dives into how modern AI models are built, trained, and scaled.
Showing all 41 articles
Blog Series
The VLM Architecture Gallery
Seventy-one architectures on one standardized plate, from gated cross-attention to encoders absorbed into the base model.
The Agent Evaluation Playbook
How to evaluate trajectories, state changes, reliability, safety, graders, budgets, and long-horizon failures.
What Autograd Actually Does
Backpropagation and autograd from scalar chain rules to Jacobian products, saved tensors, and checkpointing.
Why Transformers Do Not Explode
Initialization, residual paths, normalization, optimizer dynamics, precision, and the signatures of unstable training.
Choosing the Next Token
Temperature, search, truncation, penalties, grammars, and contrastive generation organized around entropy.
Teaching a Language Model to Follow Instructions
Supervised fine-tuning as behavior cloning: data construction, loss masking, distribution shift, and capability change.
How LoRA Changes a Model Without Rewriting It
Why low-rank updates work, how LoRA and QLoRA spend memory, and when adapters lose to full fine-tuning.
Inside an AI Agent Harness
The agent harness as a system: context assembly, action protocols, verification, containment, budgets, and event state.
Memory Is More Than Context
Working, episodic, semantic, procedural, and reflective memory as one bounded context-allocation problem.
How Multi-Agent Systems Actually Coordinate
When multiple agents beat one: diversity, specialization, topology, communication cost, orchestration, and failure modes.
Reinforcement Learning for Agents From First Principles
Reinforcement learning for agents: environments, reward design, credit assignment, horizons, and rollout systems.
How Attention Moves Information
Why attention exists, how Q, K, and V arise, and how the same operation becomes MHA, GQA, sliding windows, linear attention, MLA, and multimodal attention.
The Puzzle of Overparameterization
Interpolation, double descent and its limits, benign overfitting, implicit bias, function space, grokking, and what any of them explain about LLMs.
How LLMs Are Pretrained
Where pretraining data comes from, what survives curation, how mixtures and sequences are constructed, and how teams decide whether the recipe worked.
Mixing Data Without Losing Capabilities
Why tasks interfere, how sampling weights become loss weights, how to measure transfer, and what to do when a capability regresses.
How Language Models Learn to Use Tools
How models discover available tools at runtime, decide whether to call one, produce valid arguments, retrieve schemas, and learn from executable feedback.
The Making of an AI Agent
What changes when tool calls become a stateful loop: trajectory data, imitation versus RL, long-horizon credit assignment, memory, harnesses, and environments.
What Changed Inside the Transformer?
What changed between the original Transformer and today’s Llama, Qwen, DeepSeek, and GLM-class models—one design pressure at a time.
Scaling Laws from First Principles
Where C = 6ND comes from, why tokens per parameter is not a constant, and how to plan a large run from small experiments.
Reinforcement Learning from First Principles
Policy gradients, baselines, advantages, GAE, trust regions, and PPO derived first, then translated into language-model rollouts.
Teaching a Model What We Prefer
How comparisons become reward, why RLHF needs a KL leash, and how the same objective turns into DPO’s direct loss.
How Reinforcement Learning Teaches Models to Reason
What changes when answers can be checked automatically: verifiers, GRPO, DeepSeek-R1, exploration, and the limits of short RL runs.
Thinking in Tokens
Why a familiar Transformer can buy serial compute by writing, what reasoning traces reveal, and when longer thought makes answers worse.
How Models Improve Without Changing Their Weights
Longer chains, more samples, search, and verifiers—plus why the best inference budget depends on the prompt rather than one global setting.
How to Find What Is Slowing Your Model
A practical method for reading timelines and counters, using PyTorch Profiler and Nsight, diagnosing Roofline regimes, and testing optimization hypotheses.
The Serving Playbook
Continuous batching, paged KV, prefix caching, chunked prefill, disaggregation, distributed serving, speculative decoding, and goodput under SLOs.
The Mechanics of LLM Inference
Why inference becomes a different machine: autoregression, prefill, decode, KV cache, batching, bandwidth, latency metrics, quantization, and multimodal workloads.
The Parallelism Playbook
The six things a training workload can split, from DDP and ZeRO/FSDP through tensor, pipeline, context, expert, and hybrid parallelism.
Inside a Training Step
A first-principles walkthrough of tensor shapes, forward and backward passes, FLOPs, activation lifetimes, optimizer states, memory accounting, and checkpointing.
Talk Is Not Cheap
What changes when data is no longer local: PCIe, NVLink, NVSwitch, RDMA, topology, collective communication, and overlap.
Why Fast GPUs Still Wait for Memory
Arithmetic intensity and the Roofline model derived from first principles, then applied to GEMM, attention, fusion, optimizers, and real ML kernels.
Why GPUs Are Built for Deep Learning
Why CPUs and GPUs spend their transistor budgets differently—and how execution hierarchy, latency hiding, memory access, tiling, and Tensor Cores follow from that choice.
Inside a Modern Vision Encoder
What changed in the visual half of VLMs: intake geometry, resolution, objectives, connectors, and the encoders models actually ship.
One Transformer, Two Modalities
A block-by-block comparison: tokenizing versus patchifying, lookup tables versus projections, positions, masks, and heads.
An Image Is a Sentence
How Vision Transformers see, from the first patch embedding through ViT, DeiT, PVT, Swin, video transformers, and the cost of attention.
Across the CNNVerse
Thirty years of convolutional architecture, from sliding-window arithmetic and receptive fields to ResNet, EfficientNet, and ConvNeXt.
Evolution of ML Architectures
Sixty-seven years of neural architectures, including how information moves, what training costs, what inference carries, and how gradients flow backward.
How Models Know Where They Are
Absolute, sinusoidal, relative, RoPE, NoPE, 2D and Fourier features, and multimodal M-RoPE—explained visually.
What an Optimizer Actually Does
A mechanical tour from SGD to Muon: how optimizers build an update, what their state costs, and where a training step stalls.
The Geometry of Normalization
Every normalization layer from BatchNorm to Peri-LN, plus why the modality interface in a VLM is a normalization problem.
Why Neural Networks Need Nonlinearity
From sigmoid and ReLU to GELU, SwiGLU, and squared ReLU—with animated derivations and a map of what modern models use.