How LoRA Changes a Model Without Rewriting It

The ledger nobody reads

Every conversation about parameter-efficient fine-tuning starts in the wrong place. It starts with "LoRA is cheaper." That is true but it explains nothing, because it does not say which cost went away, and until you know that, you cannot predict when the trick will fail. So before any low-rank matrices, here is the accounting.

Take a 7B-parameter model and fine-tune it the usual way: bf16 weights, AdamW, mixed precision. Count the bytes per parameter that have to live in GPU memory simultaneously.

• Weights — 2 bytes (bf16).
• Gradients — 2 bytes, one per weight, produced by the backward pass.
• Master weights — 4 bytes, an fp32 copy so that tiny updates are not annihilated by bf16's ~8 bits of mantissa.
• Adam first moment m — 4 bytes.
• Adam second moment v — 4 bytes.

Sixteen bytes per parameter. For 7B that is 112 GB before a single activation is stored, before any parallelism overhead, before the KV of a long sequence. An 80 GB H100 cannot hold it. The model itself — the thing you actually care about — is 14 of those 112 gigabytes. Seven-eighths of the memory is scaffolding for the optimizer.

That ratio is the whole story. Full fine-tuning is expensive not because models are big, but because the machinery of changing a number is roughly seven times heavier than the number itself. If you could arrange for the vast majority of parameters to never change, all seven-eighths would evaporate and you would be left carrying the model as dead weight — read-only, cheap, shardable.

There is a second ledger, and in production it hurts more. A fine-tuned model is a full copy. Hosting fifty customer-specific variants of a 70B model means fifty times 140 GB of checkpoints, fifty cold starts, fifty things to version. The LoRA paper opens with exactly this complaint about GPT-3: deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. The storage bill, not the training bill, is what made the problem urgent.

Figure 1. The optimizer-state stack for a 7B model, built up byte by byte, then dismantled by freezing and by quantization. Bar heights are exact arithmetic from bytes-per-parameter; activation memory is shown as a representative band because it depends on batch, sequence length and recomputation policy, not on the method. The final column is the QLoRA regime and is previewed here; it is derived in § QLoRA.

If you want the full picture of what a training step is spending — the activation stashes, the recomputation trade, the all-reduce — that is the subject of Inside a Training Step and The Parallelism Playbook. This article is about the one lever those posts do not pull: making almost every parameter read-only.

Fine-tuning is learning a delta

Write down what fine-tuning does to a single weight matrix. You start from a pretrained W and you end somewhere else:

This is not a modelling assumption. It is a tautology — any final weight matrix can be written as the initial one plus a difference. But writing it this way changes what question you ask. Instead of "what should the weights be," you ask "what is the change made of."

For a d_out × d_in matrix, full fine-tuning lets ΔW be an arbitrary element of ℝ^d_out×d_in. It has as many degrees of freedom as the weight matrix itself. Every parameter-efficient method is an answer to one question:

Different answers give different methods. ΔW is sparse gives diff-pruning and BitFit. ΔW is diagonal, or a rescaling of activations, gives IA³. ΔW is low-rank gives LoRA. Some methods do not touch W at all and instead change what flows through it — that is prompt and prefix tuning, which we will fold into the same picture later.

An analogy that survives contact with the math

Imagine a spreadsheet of salaries: one row per employee, one column per pay component. Suppose the company gives everyone a raise. The naive way to record it is a second spreadsheet of the same size — every cell gets its own number. That is full fine-tuning.

But most real raises are not arbitrary. A common one is "each department gets a multiplier, each pay component gets a weight, and the raise in a cell is the product." That is a rank-one change: instead of d_out × d_in numbers you store d_out + d_in. Allow r such patterns superimposed and you have a rank-r change: r(d_out + d_in) numbers.

The compression is not free — the family of rank-r changes is a thin sliver of all possible changes. The bet is that whatever "adapting to a task" means, it looks more like a handful of superimposed patterns than like arbitrary noise. That bet needs evidence, so let us go find some.

Why the delta might be small

Two results from before LoRA existed set up the whole argument.

Li et al. (2018) asked how many effective dimensions an optimization problem has. They took a network with D parameters and refused to train it in ℝ^D. Instead they fixed a random projection P from a small space ℝ^d into ℝ^D and trained only z:

The smallest d that reaches, say, 90% of full-training performance is the intrinsic dimension of the objective. It was consistently far smaller than D.

Aghajanyan, Zettlemoyer and Gupta (2020) ran that probe on pretrained language models and got a striking number: by optimizing only 200 trainable parameters randomly projected back into the full space, we can tune a RoBERTa model to achieve 90% of the full parameter model performance on MRPC. Two hundred numbers. They also found that intrinsic dimension shrinks with better and longer pretraining — a larger, better-trained model is easier to steer, not harder.

The interpretation matters and it is easy to overstate. This is not evidence that the fine-tuning solution is simple. It is evidence that the set of acceptable solutions is large and sprawling enough that a random low-dimensional slice through weight space intersects it. Pretraining does not leave the model at a needle-point optimum; it leaves it inside a broad basin whose floor is wide in most directions. Fine-tuning is a short walk across that floor, not a climb to a new peak.

LoRA takes one further step, and it is a step of faith rather than of proof. Low intrinsic dimension says ΔW lives near some low-dimensional manifold. LoRA asserts that a good choice of manifold is low matrix rank, per weight matrix. The paper is candid that this is a hypothesis: we hypothesize the updates to the weights also have a low intrinsic rank during adaptation.

Rank, from first principles

Rank gets taught as "the number of linearly independent rows," which is correct and useless. Here is the version that makes LoRA obvious.

A matrix M ∈ ℝ^m×n is a map from ℝⁿ to ℝ^m. Its rank is the dimension of the image: how many independent directions the output can actually occupy, no matter what you feed in. A rank-r matrix, whatever its shape, only ever produces outputs inside an r-dimensional subspace, and it only ever looks at an r-dimensional subspace of its input. Everything else it annihilates.

Every rank-r matrix factors through ℝ^r. That is the entire content of the LoRA parameterization:

Read the factorization as a two-step protocol. Ax asks r questions about the input — "how much of pattern 1 is present, how much of pattern 2" — and discards everything else. B takes those r answers and adds a corresponding mixture of r output patterns to the residual stream. The rank is literally the number of wires in a bottleneck soldered in parallel to a frozen circuit.

The parameter arithmetic

For a square d × d matrix the ratio is 2r/d. With d = 4096 and r = 8 that is 0.39%. The break-even point — where a low-rank factorization stops saving anything — is r = d/2. If you find yourself reaching for rank 2048 on a 4096-wide model, you have not chosen a cheap method; you have chosen an expensive one with extra steps.

Which rank-r matrix is best?

There is a clean answer for approximating a matrix you already have. Write the singular value decomposition M = UΣV^⊤, with singular values σ₁ ≥ σ₂ ≥ … The Eckart–Young theorem says the best rank-r approximation in Frobenius norm is the truncation that keeps the top r singular triplets, and the error it leaves behind is exactly the tail of the spectrum:

So "is this matrix low-rank?" is really "does its spectrum decay fast?" A spectrum that plunges means a handful of patterns carry everything. A flat spectrum means every direction matters equally and no truncation is safe. Figure 2 makes this tangible: watch the reconstruction and the spectrum at the same time and you will never again need to be told what rank means.

Figure 2. A 24×24 matrix rebuilt one singular triplet at a time. Left: the running rank-r reconstruction. Centre: the residual, on the same colour scale. Right: the spectrum, with kept components lit and the discarded tail dimmed; the number under it is the exact Eckart–Young error. Toggle Slow decay to see a matrix that refuses to compress — the residual stays loud until r is nearly full. The matrices are synthetic, generated in-page from a fixed seed so the arithmetic is exact rather than illustrative.

Two lessons carry forward. First, rank is a property of the spectrum, not of the size — a 4096×4096 matrix can be rank 3. Second, and this is the part people skip: LoRA does not truncate a ΔW it already has. It never computes the full update. It constrains the search to the rank-r manifold from step one and lets gradient descent find the best point on it. Those are different problems, and the second has no Eckart–Young guarantee attached. Whether SGD on the factored form finds the optimal rank-r update is an empirical question, not a theorem.

LoRA

The method is three lines. Freeze W. Add a parallel low-rank branch. Scale it.

Note that W and BA see the same input and their outputs are summed coordinate-wise. The branch is parallel, not sequential. That single design choice is the reason LoRA has no inference cost and adapters do, and we will come back to it.

Why B = 0 and A random

There are four ways to initialize a product of two matrices and only one of them is right.

• Both random. Then ΔW ≠ 0 at step zero. You begin by injecting noise into a model that was expensively trained not to have any. Loss spikes, and the first phase of fine-tuning is spent undoing the damage.
• Both zero. Then ΔW = 0, which is what we want, but look at the gradients below: both are proportional to the other matrix. Zero times zero stays zero forever. The branch is dead on arrival.
• A = 0, B random. Works, by symmetry, and some implementations do it. It changes which matrix moves first.
• B = 0, A random. The standard choice. ΔW = 0 at initialization, so the adapted model is exactly the pretrained model — a strict function-preserving start — and the gradient path is alive.

This "identity at initialization" principle is not unique to LoRA. Houlsby's adapters use near-identity initialization for the same reason. Flamingo's cross-attention layers are gated by a tanh whose gate starts at zero. Whenever you bolt something onto a pretrained network, you want it to be invisible on the first forward pass.

The backward pass, written out

Let s = α/r and let g = ∂ℒ/∂h be the gradient arriving at the layer output. Then:

Three things fall out of these expressions that are worth sitting with.

On the very first step, only B moves. With B = 0, the gradient of A is exactly zero. B takes the first step; only then does A receive signal. LoRA training has a brief asymmetric warm-up baked into its initialization, and it is one of the reasons the two matrices behave differently throughout training — which is the observation LoRA+ later turned into a method.

The gradient still flows all the way down. ∂ℒ/∂x contains W^⊤g. Freezing a weight does not let you skip its backward pass; you still need the full chain rule down to the earliest adapted layer. LoRA saves optimizer memory, not backward compute. If you adapt only the top eight layers you genuinely can truncate the backward pass — but nobody does that, because it works badly.

Some activation stashes disappear. For a frozen linear layer with no adapter, the backward pass needs W to compute ∂ℒ/∂x, but it never needs x, because there is no ∂ℒ/∂W to compute. Autograd can drop that input activation entirely. Adapted layers must still stash x, since ∂ℒ/∂A depends on it. So adapting every linear layer gives back most of that saving — a small, real, rarely-mentioned cost of "target all modules."

Figure 3. The camera follows a single activation vector through a frozen projection and its parallel rank-r bypass, then reverses to follow the gradient back. Stages 8–11 zoom into the two weight gradients; stage 12 shows what the optimizer actually has to hold; stages 13–14 fold the branch back into the frozen matrix for deployment.

What this buys, in numbers

On GPT-3 175B the original paper reports the two savings separately, and they are different sizes. Training VRAM went from 1.2 TB to 350 GB — a factor of about 3.4, because the frozen weights are still there and still 350 GB. The checkpoint went from 350 GB to 35 MB — a factor of 10,000 — because the frozen weights are not part of the checkpoint at all. Throughput improved by about 25%. That is a modest speedup, and it is the honest one: you have removed the optimizer update for 99.99% of parameters, not the matrix multiplies.

Figure 4 lets you turn the dials yourself on a Llama-2-7B-shaped model.

Figure 4. Exact parameter arithmetic for Llama-2-7B geometry (d = 4096, 32 layers, SwiGLU MLP with hidden width 11008, 6.74 B parameters total). Rank sweeps 1 → 512; the dashed line marks the point where the adapters cost more than the weights they are adapting. Training-memory figures assume 16 bytes per trainable parameter (fp32 master, m, v, gradient) and 2 bytes per frozen parameter, and exclude activations.

Scaling, initialization, and the α/r question

The factor α/r looks like an afterthought. It is the most consequential hyperparameter in the method and the least understood.

What the scaling is for

Consider what happens to BAx as you increase r with a fixed per-entry initialization scale for A. The product sums r rank-one contributions. More terms, larger typical magnitude. Without compensation, doubling the rank changes the effective size of the update, and every learning rate you had tuned is now wrong.

Dividing by r is the crude fix. The paper is explicit about its purpose — this scaling helps to reduce the need to retune hyperparameters when we vary r — and equally explicit that α is not really a separate knob: tuning α is roughly the same as tuning the learning rate. In practice people fix α and vary r, which silently changes the effective step size for the adapter, which is exactly the trap.

The rank-stabilization correction

Kalajdzievski (2023) did the large-r analysis properly. Asking that neither the forward activations nor the backward gradients through the adapter blow up or vanish as r grows gives a different exponent:

The consequence is uncomfortable. Under α/r, the gradient reaching the adapter shrinks as rank grows, so higher-rank runs learn more slowly and converge to roughly the same loss as low-rank ones. That is precisely the empirical pattern everyone cited as evidence that "rank doesn't matter beyond 8." Under α/√r, larger ranks keep improving. As the rsLoRA write-up puts it, the conventional scaling slows learning for ranks outside this very small rank regime, which has fostered a misconception about how low-dimensional adaptation really is.

Asymmetric learning rates

LoRA+ makes a related width argument: because A maps from width d into width r and B maps back out, giving them the same learning rate is not scale-correct for large d. Using a fixed ratio η_B/η_A ≫ 1 recovers efficient feature learning, and the reported gains are 1–2 points with up to a 2× speedup at identical cost. This is the same asymmetry the backward-pass equations already hinted at.

Smarter initialization

Random-plus-zero is the default, not the optimum. PiSSA initializes A and B from the principal singular components of W itself and freezes the residual, so the trainable subspace starts aligned with the directions the pretrained weight already considers important. MiLoRA does the opposite, using the minor components, on the theory that the important directions should be left alone. Both beat random initialization on their own benchmarks, which is a good reminder that "which subspace" is a real design choice and not a detail.

The defaults that actually work

• α = 2r is the most defensible default; it is what Biderman et al. recommend after a systematic sweep.
• Turn on rank-stabilized scaling if you intend to use r > 32.
• LoRA learning rates live around 1e-4 to 5e-4 — roughly 10–50× higher than full fine-tuning, because the adapter starts at zero and has to travel.
• Dropout on the adapter helps small models and stops helping around 30B; QLoRA used 0.1 up to 13B and 0.05 above.

Which matrices, and how much rank each

A transformer block has seven linear maps worth adapting: W_q, W_k, W_v, W_o in attention, and gate / up / down in the MLP. They are not equally sized and they are not equally useful.

Where the parameters live

For Llama-2-7B geometry, per layer: attention holds 4d² = 67.1 M parameters and the SwiGLU MLP holds 3 × d × 11008 = 135.3 M. Two-thirds of the adaptable weight mass is in the MLP. Any argument that stops at the attention projections is ignoring most of the model. This is not an accident of one architecture — the 4×-ish MLP expansion has been standard since the original transformer.

The original ablation, and what it actually said

The LoRA paper ran a clean experiment: fix a budget of 18 M trainable parameters on GPT-3 175B across all 96 layers, then spend it different ways. Because the budget is fixed, adapting more matrix types means a lower rank each.

Read the columns, not the winner. Rank 2 across four matrices beats rank 8 on one. The paper's own conclusion: it is preferable to adapt more weight matrices than adapting a single type of weights with a larger rank. Spread the budget thin and wide.

There is a second, subtler reading. W_k is the weakest place to spend. That is not mysterious: queries and keys enter attention only through the bilinear form q^⊤k, so a perturbation of W_q and a perturbation of W_k have overlapping effects, and adapting both spends budget twice on one degree of freedom. W_v and W_o, by contrast, sit on the value path and directly determine what gets written back into the residual stream.

Modern practice has moved past the table. QLoRA targets all linear layers of the base model; Biderman et al. recommend the same after finding that restricting to attention consistently underperforms. The 2021 result was about spending a tiny budget well. Once the budget is large enough to include the MLP, include the MLP.

Figure 5. A transformer block with the seven adaptable projections, sized by parameter count. The camera walks the fixed-18 M-budget ablation from Hu et al. (2021) Table 5 — those accuracy bars are the published values, not illustrations — then pans to the MLP to show the two-thirds of weight mass that the 2021 study excluded by design.

Should every matrix get the same rank?

Uniform rank is a convenience, not a principle. AdaLoRA makes the allocation adaptive by parameterizing the update in SVD form, ΔW = PΛQ, and pruning singular triplets with low importance scores during training. The parameterization is the clever part: pruning a diagonal entry of Λ costs one number and needs no re-factorization, so you can reallocate rank continuously without ever running an exact SVD.

In practice AdaLoRA's gains show up mainly in low-budget regimes, which makes sense — allocation only matters when the budget binds. If you can afford rank 64 everywhere, allocate uniformly and spend your time on data instead.

Merging, swapping, serving

Because the branch is parallel and linear, the deployed model can absorb it exactly:

This is the property adapters cannot have. A bottleneck adapter is applied after the sublayer, so its computation is sequential — an extra pair of small matrix multiplies that cannot be fused away and that the GPU cannot hide behind other work. The LoRA paper measured it on GPT-2 medium: at batch size 1 and sequence length 128, the Houlsby adapter added 30.3% to forward latency and the leaner variant added 20.7%, while LoRA added nothing by construction. At batch 32 and sequence 512 the penalty nearly vanishes, because now there is enough parallel work to saturate the hardware. Adapter latency is an online-serving problem, not a training problem. If your batches are large, the objection evaporates.

Unmerging and its hazards

Merging is reversible: subtract to get W back, add a different adapter, and you have hot-swapped tasks with one matrix operation. Two cautions.

First, precision. Merge and unmerge in bf16 repeatedly and rounding error accumulates in W; the base you recover is not the base you started with. Keep an authoritative copy, or do the arithmetic in fp32.

Second, quantization. Merging a bf16 adapter into an NF4 base is lossy in a way that is easy to miss: you must dequantize, add, and re-quantize, and the re-quantization error can be comparable to the update you just added. The usual answers are to merge into a full-precision base and quantize afterwards, or to keep the adapter unmerged at inference and pay the small parallel-branch cost.

Many adapters, one base

The deployment story that made LoRA an industry default is not single-task fine-tuning; it is fleets. One 70B base in VRAM, hundreds of 40 MB adapters in host memory, and a router. Because the adapters are unmerged, a single batch can contain requests for different tasks: the base matmul is a normal GEMM over the whole batch, and the low-rank branch is a grouped operation over per-request adapters. S-LoRA works this problem properly, with a unified paged memory pool shared between adapter weights of differing ranks and KV-cache blocks of differing lengths, plus custom kernels for the heterogeneous batching — reporting thousands of concurrent adapters on a single GPU. The memory-management ideas there are close cousins of the ones in The Serving Playbook.

Figure 6. Left half: the parallel branch folded into the frozen matrix, then unfolded and replaced with a different task's adapter. Right half: the unmerged regime, where one base GEMM serves a mixed batch and each request's low-rank branch is gathered separately. The latency bars at stage 5 are the measured GPT-2-medium numbers from Hu et al. (2021), Table 1.

Composing adapters

Adding two adapters' deltas — W + Δ₁ + Δ₂ — sometimes composes their behaviours and sometimes produces something worse than either. There is no guarantee: the two low-rank subspaces were found independently and nothing made them orthogonal. Weighted merging, orthogonality constraints and learned routing over adapter libraries are all active work, and none of it is settled enough to be a default. If you need two behaviours reliably, the boring answer — train one adapter on the union of both datasets — remains the strongest baseline.

QLoRA: training through a compressed model

LoRA leaves you carrying the frozen weights at 2 bytes each. For a 65B model that is 130 GB — still two GPUs. The next question writes itself: if the base never changes, why store it at full precision at all?

QLoRA answers it, and the headline is that a 65B model becomes finetunable on a single 48 GB GPU while preserving full 16-bit finetuning task performance. Three ingredients, each solving a distinct problem.

Quantization, from first principles

A b-bit quantization scheme is a codebook of 2^b real values plus a rule for assigning each weight to one of them. The design question is: where should the codewords go?

Uniform integer quantization spaces them evenly across [−absmax, absmax]. That is optimal if the weights are uniformly distributed. They are not. Neural network weights are approximately zero-centred Gaussian, so most of the probability mass sits in a narrow band around zero while the range is set by rare outliers. Uniform bins therefore spend most of their 16 codewords in the tails, where there are almost no weights, and leave the crowded centre coarsely resolved. You are allocating precision inversely to demand.

The fix is old and clean. Quantile quantization places bin boundaries at the quantiles of the data distribution, so each bin receives equal probability mass. Equal mass per code means the discrete code has maximum entropy, which is the information-theoretic definition of not wasting bits.

NF4 — 4-bit NormalFloat — is quantile quantization for a standard normal, with two engineering touches: the levels are normalized so the codebook spans [−1, 1], and the split between negative and positive levels is made asymmetric so that zero is exactly representable. Exact zero matters more than it sounds; padding, masks and genuinely-zero weights should not acquire a small bias. QLoRA's claim is that this is information theoretically optimal for normally distributed weights, and empirically NF4 beats both FP4 and INT4.

To use a distribution-matched codebook you must first put the weights on that distribution's scale. QLoRA normalizes blockwise: split the tensor into blocks of 64, divide each block by its own absmax, quantize the normalized values. Blocking is what makes it robust — a single outlier weight can only corrupt the resolution of its own 64 neighbours instead of the entire tensor. That matters because large-model activations and weights genuinely do contain systematic outliers, which is the same phenomenon that forced mixed-precision decomposition in LLM.int8().

Figure 7. A Gaussian weight distribution with 16 codewords placed two ways. The quantile levels are computed in the page from an inverse-normal approximation rather than quoted, so the geometry is real; the published NF4 codebook is constructed this way with a small asymmetry that makes zero exact. Stages 7–9 zoom into a single 64-weight block to show why blockwise absmax normalization contains outlier damage; stages 10–12 build the bit ledger for double quantization.

Double quantization: quantizing the quantizers

Blockwise scaling has a cost that is easy to overlook. One fp32 absmax constant per 64 weights is 32/64 = 0.5 extra bits per parameter — on top of a 4-bit payload, a 12.5% overhead. So QLoRA quantizes the constants too: the absmax values are themselves 8-bit quantized in blocks of 256, with a second-level fp32 constant per block of blocks.

Effective storage becomes about 4.127 bits per parameter. For a 6.74 B model that is 3.48 GB instead of 13.5 GB. The paper reports that double quantization costs nothing in accuracy, which is the expected outcome — you are compressing a second-order statistic, not the signal.

The gradient path

Here is the part that confuses people. NF4 is a storage format, not a compute format. The forward pass dequantizes each weight block to bf16 just before its matrix multiply and throws the result away afterwards. So:

The base receives no gradient, so its quantization error is never re-introduced or accumulated. It is a fixed, deterministic perturbation of the frozen function. And crucially, the adapter is trained on top of the already-quantized model — so whatever the quantization broke, gradient descent has the whole run to compensate for it. This is the most plausible reading of why 4-bit inference on an unmodified model degrades measurably while 4-bit adapter fine-tuning does not; the paper poses exactly this question and answers it empirically, finding that NF4 with double quantization matches bf16.

The costs are real and rarely stated: you pay a dequantization kernel on every forward and every recomputation, so QLoRA is slower per step than bf16 LoRA. You are trading time for memory. Trading in the other direction — spending memory to save time — is the subject of How to Find What Is Slowing Your Model.

Paged optimizers

The last ingredient is unglamorous plumbing. With gradient checkpointing and long sequences, memory usage is spiky: most of the time you are fine, and then one recomputation boundary asks for a few extra gigabytes and the process dies. Paged optimizers put the optimizer state in NVIDIA unified memory so those pages can be evicted to CPU during a spike and paged back afterwards, turning a hard OOM into a slowdown. It is the same idea as OS virtual memory, applied to the one allocation that can tolerate the latency.

DoRA: magnitude and direction

DoRA begins with a diagnostic rather than a method. If LoRA and full fine-tuning reach different quality, they must be moving weights differently. How?

Decompose each weight matrix into a per-column magnitude and a unit-norm direction:

Every column of a weight matrix is now "a direction, times a loudness." Track how much each changes over a fine-tuning run — call them ΔD and ΔM — and plot one against the other. The paper's finding is a difference in sign: across checkpoints, full fine-tuning shows a negative relationship between magnitude change and direction change, while LoRA shows a positive one. LoRA turns the two knobs together; full fine-tuning trades them off, and in particular can make substantial direction changes with only slight magnitude changes.

Why would that be? Because in W + BA, magnitude and direction are not separate parameters. Adding a vector to a column changes both, coupled by geometry. To rotate a column without lengthening it, the low-rank update must produce a component almost perpendicular to the existing column of exactly the right size — and it has only r degrees of freedom to arrange that for every column simultaneously. Full fine-tuning has d_out free numbers per column and no such constraint.

DoRA's fix is to give the magnitude back its own parameter:

The normalization strips whatever length change the low-rank update happened to cause, and m — one number per output column, a rounding error in the parameter budget — sets the length explicitly. The rank is now spent purely on direction, which is what it was scarce for.

Figure 8. Left: a single weight column in 2-D, updated additively (length and angle move together) and then in DoRA's parameterization (normalize, then rescale). Right: the ΔD–ΔM scatter that motivated the method. The scatter is a schematic reproduction of the sign structure reported in Liu et al. (2024) Figure 1 — the slopes and clustering are drawn to match the published qualitative pattern, not the published values.

Practicalities. DoRA merges to a single matrix at deployment, so inference cost is unchanged. Training cost is not: the denominator depends on BA, so column norms must be recomputed every step and differentiated through. It is supported in Hugging Face PEFT behind use_dora=True, and the reported gains are consistent but modest — reliably better than LoRA at equal rank on commonsense reasoning, visual instruction tuning and image/video-text tasks, which is a meaningful result precisely because it is unglamorous.

The wider family

LoRA won, but it did not arrive first and it is not the only shape. It helps to see the whole family sorted by what they modify.

Additive modules: adapters

Houlsby et al. (2019) inserted small bottleneck MLPs after each sublayer:

Same bottleneck, same near-identity initialization, one crucial difference: the residual is around the adapter, so the computation is sequential. On GLUE they came within 0.4% of full fine-tuning while training 3.6% of parameters — an excellent result that was, in 2019, mostly about storage. The sequential form is why they cannot be merged and why they cost online latency.

Input-space methods: prefix and prompt tuning

A different move entirely: leave every weight alone and change what the model attends to.

Prefix tuning (Li & Liang, 2021) prepends p trainable key/value vectors to the attention of every layer. Every token can attend to this virtual context; nothing in it comes from the input. Parameters: 2Lpd. Training is notoriously finicky and typically needs a reparameterizing MLP to be stable.

Prompt tuning (Lester et al., 2021) is the minimal version: p trainable embeddings at the input layer only. Parameters: pd — tens of thousands. The paper's title is the finding: it is the power of scale. At small model sizes prompt tuning is far behind full fine-tuning; the gap closes as models grow toward 10B. Continuous prompts are gradient-descent's version of prompt engineering, and like prompt engineering they work better on stronger models.

Both share a structural cost the LoRA paper called out: the prefix consumes sequence length, so it competes with the actual task input for context.

Rescaling: IA³

IA³ learns three vectors per block and multiplies them elementwise into the keys, the values, and the MLP's intermediate activations:

This is ΔW constrained to be a diagonal rescaling — the most extreme structural restriction in common use, and it merges cleanly into the adjacent weight matrix because scaling rows of K is the same as scaling rows of W_k. On the T-Few benchmark it beat in-context learning at a fraction of the inference cost, which was the paper's actual point.

Selective: BitFit

BitFit trains only the bias vectors — under 0.1% of parameters for BERT, and competitive with full fine-tuning on small-to-medium data. Its interest today is mostly conceptual: the authors argue the result supports the view that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge, which is the same claim the intrinsic-dimension literature makes from the other direction. Note the architectural catch: Llama-style models have no biases in their linear layers at all, so BitFit is simply inapplicable to most current LLMs.

One frame for all of them

He et al. (2022) showed these are not seven unrelated tricks. Every one of them computes a modification Δh to some hidden state and differs along three axes: the function that produces Δh (low-rank, bottleneck MLP, attention over learned vectors, elementwise scale), the insertion form (parallel or sequential), and the composition (add, gate, or multiply). Prefix tuning, read through this lens, turns out to be a gated additive modification of the attention output — structurally an adapter wearing a costume.

Figure 9. The same block, re-marked for each method: what is frozen, what is trainable, and where the modification enters. Parameter counts are computed in-page from Llama-2-7B geometry with the stated hyperparameters (adapter bottleneck 64, prefix and prompt length 32, LoRA rank 8 on all linear layers) and are exact for those settings. The merge/latency badge records whether the method can be folded into the frozen weights.

PEFT versus full fine-tuning

For three years the community answer was "LoRA matches full fine-tuning." The careful studies say something more interesting and more useful.

Learns less

Biderman et al. (2024) compared the two on Llama-2-7B across programming and mathematics, in both instruction-tuning (~100K pairs) and continued-pretraining (~20B tokens) regimes. In standard low-rank settings LoRA substantially underperformed. On code continued pretraining the best LoRA configuration at 20B tokens roughly matched full fine-tuning at 4B tokens — a 5× sample-efficiency gap. On instruction tuning LoRA could close the gap, but only at rank 256, far above the 8–64 that everyone actually runs.

They also measured the mechanism: full fine-tuning learns perturbations with rank 10–100× greater than typical LoRA configurations. Whether that causes the gap or merely accompanies it is not settled, but it is the most direct evidence available that the low-rank hypothesis is a real constraint on hard tasks rather than a free lunch.

Forgets less

The same study found the mirror image. Measuring source-domain ability (HellaSwag, ARC-Challenge, WinoGrande) after training on a target domain, LoRA degraded less. On code continued pretraining at 20B tokens, full fine-tuning scored 0.545 on that average while LoRA at rank 256 scored 0.617; on code instruction tuning, 0.414 versus 0.509 for LoRA at rank 64. LoRA also preserved base-model behaviour better than weight decay or dropout did, and maintained more diverse generations.

So there is a frontier, not a winner: rank buys target-domain performance and spends source-domain retention. Where you want to sit depends entirely on whether "the model got worse at everything else" is a bug or an irrelevance for your deployment.

The solutions are not the same shape

Shuttleworth et al. (2024) looked at the resulting weight matrices spectrally and found something sharper. LoRA-tuned matrices acquire what they call intruder dimensions: high-ranking singular vectors that are approximately orthogonal to every singular vector of the pretrained matrix. Fully fine-tuned matrices almost never contain them — full fine-tuning makes small adjustments to the directions that were already there. The count is rank-dependent and vanishes again at very high rank, where LoRA stops behaving low-rank at all.

The causal experiment is the interesting bit: scaling down the singular values associated with intruder dimensions after training improves modelling of the pretraining distribution with minimal cost to the downstream task, which is direct evidence that these directions carry the forgetting. And the effect compounds under sequential fine-tuning, which is bad news for anyone stacking adapters over time.

Figure 10. The trade-off as a frontier. The four labelled source-domain retention values (full fine-tuning and LoRA, code CPT and code IFT) are the published numbers from Biderman et al. (2024); the target-domain axis and the curve joining the rank points are schematic, drawn to show the shape of the trade-off rather than to reproduce measurements. Stage 7 overlays the spectral picture from Shuttleworth et al. (2024) and is likewise a qualitative illustration.

The rule that actually predicts

Every result above points the same way. PEFT works when the target behaviour is a re-weighting of capabilities the base model already has — format, tone, task framing, domain vocabulary it has already seen, tool syntax. PEFT struggles when you are adding capability the base model does not possess — a new language, a genuinely unfamiliar formalism, twenty billion tokens of a distribution it never saw. Ask which of those you are doing before you pick a method, and the method mostly picks itself.

Worth noting for completeness: LoRA is not fundamentally less expressive. The original paper observes that applying LoRA to every matrix with r equal to the full rank recovers the expressiveness of full fine-tuning, and Zeng and Lee (2024) give conditions under which a sufficiently deep low-rank-adapted network can represent any target model. The constraint is a budget, not a wall.

PEFT in vision-language models

Everything above was written about a transformer stack. VLMs are where the ideas get interesting, because a VLM is not one model — it is a frozen vision tower, a small connector, and a language model, and each of the three wants a different treatment.

The standard recipe is already PEFT

LLaVA-1.5's two stages look like this. Stage 1 freezes both the CLIP vision encoder and the language model and trains only the two-layer MLP projector on 558K image–text pairs. Stage 2 keeps the vision encoder frozen and trains the projector plus the language model on a 665K instruction mixture.

Stage 1 is selective PEFT by any definition: over 99% of parameters are frozen and the trainable module is a handful of megabytes. Nobody calls it that, because the framing was "modality alignment" rather than "efficiency," but the mechanics are identical — a small trainable bridge between frozen towers. BLIP-2's Q-Former is the same idea with a heavier bridge; Flamingo's gated cross-attention layers are the same idea implemented as adapters interleaved into a frozen language model, complete with a tanh gate initialized to zero so the augmented model starts exactly equal to the original. The identity-at-init principle from § LoRA shows up independently everywhere.

Where LoRA goes, and where it must not

The LoRA variant of stage 2 replaces full language-model training with adapters and keeps the projector fully trainable. That asymmetry is the important design decision and it follows directly from the theory:

• The language model is being adjusted. It already knows how to write; you are re-weighting behaviours it has. Low-rank is the right constraint, and it protects the text-only abilities you would otherwise burn.
• The projector is being created. It is learning a map that does not exist yet, from vision-encoder feature space into token-embedding space. There is no pretrained function to perturb, so "small update to a good initialization" is not a meaningful description of the problem. Train it fully. It is small enough that this costs nothing.
• The vision tower stays frozen unless the domain has genuinely shifted — documents, medical imaging, remote sensing, anything CLIP's pretraining did not cover. Then LoRA on the ViT is the usual first move, at low rank, before considering unfreezing.

Ranks in VLM adaptation run higher than in text: r = 128 with α = 256 is a common working configuration, and a differential learning rate for the projector (around 2e-5) versus the adapters (1e-4 to 2e-4) is standard. That is not superstition; it reflects the two-jobs-at-once structure above.

Two hazards specific to VLMs

First, the memory arithmetic changes. A single image at 336px through CLIP ViT-L/14 becomes 576 tokens, and tiling schemes multiply that. Sequence length — and therefore activation memory — dominates in a way it does not for text fine-tuning. LoRA removes optimizer state; it does not remove activations. On a VLM run you can find yourself with a comfortable weight budget and an OOM anyway, and the fix is gradient checkpointing and sequence packing, not a lower rank.

Second, an adapter cannot repair a broken interface. If the vision tower's resolution, patch grid or normalization does not match what the projector expects, no amount of low-rank adaptation downstream will fix it — you will get a model that trains to a plausible loss and answers image questions from language priors. Check the interface before you tune it.

When not to use LoRA

The useful version of this section is a set of triggers, not a disclaimer.

1. You are adding knowledge, not adjusting behaviour. New language, new formalism, a domain the base model has genuinely never seen. Biderman's continued-pretraining results are unambiguous here: low-rank is sample-inefficient exactly when the distribution shift is large.
2. You are training on more than a few billion tokens. The gap widens with data. If you can afford the tokens you can usually afford the optimizer states.
3. You need to change the embedding table or tokenizer. Adding vocabulary means new rows, which are new parameters, not low-rank perturbations of old ones. Train the embeddings.
4. Your bottleneck is step time, not memory. LoRA's speedup is modest and QLoRA is actively slower per step. If you have the memory, full fine-tuning at the same wall-clock budget will usually learn more.
5. Your rank is drifting above d/2. At that point the factorization stores more numbers than the matrix it is approximating. Either the task needs full fine-tuning or your scaling is wrong — check § Scaling before you conclude the former.
6. You are stacking adapters sequentially over months. Intruder dimensions accumulate under sequential fine-tuning, and the retention advantage erodes. Periodically retrain from the base on the union of your data.
7. The problem is not a fine-tuning problem. A retrieval system, a better prompt, or a few well-chosen examples solves a surprising fraction of what people reach for LoRA to fix, at a fraction of the operational cost. Fine-tuning changes behaviour; it is a poor way to install facts.

A default that works

If you want one configuration to start from and adjust only when something forces you to: LoRA on all linear layers, rank 16, α = 32, rank-stabilized scaling on, dropout 0.05, learning rate 2e-4 with a warmup, and the base in bf16 if it fits and NF4 if it does not. Then sweep rank, and only rank, and watch both the target metric and one held-out source-domain benchmark. If the target metric is still climbing at rank 128, you have learned something important — probably that you should not be using LoRA for this.

References

Grouped by topic. Every entry links to its primary source.