LoRA: Low-Rank Adaptation of Large Language Models
TL;DR
Freeze the big model. Add tiny low-rank matrices to each weight. Train only those. Get ~10,000× fewer trainable params, same quality as full fine-tuning, and the ability to swap thousands of fine-tunes from one base model.
Why it matters
Before LoRA, fine-tuning a 70B model needed an enterprise-scale cluster. After LoRA, it fits on a single 24GB consumer GPU (with QLoRA). The entire 'custom LLM' industry — every fine-tune marketplace, every domain-specific assistant, every multi-tenant LLM serving platform (S-LoRA, Punica, Lorax) — exists because of this paper.
The Key Idea
Weight updates during fine-tuning have low intrinsic rank. So instead of updating the full W₀ (which is huge), parameterize ΔW = B·A where B is d×r and A is r×k for a tiny rank r (often 4-16). Train only B and A. Reconstruct W = W₀ + BA at inference.
How it works
1. The Math
For a frozen weight matrix W₀ ∈ ℝ^(d×k), define the fine-tuned weight as W = W₀ + ΔW where ΔW = BA, with B ∈ ℝ^(d×r) and A ∈ ℝ^(r×k). The rank r is tiny — often 4, 8, or 16. Total trainable params: r·(d+k) instead of d·k. For r=8, d=k=4096, that's a 256× reduction.
2. Initialization Trick
Initialize A with small random values and B with zeros. Then BA = 0 at the start, so the model's initial output exactly matches the base model — no warmup needed. The model 'learns the adapter from zero' without first having to undo a random perturbation.
3. Where to Apply It
The paper studied applying LoRA to Q, K, V, output, and the FFN. Result: applying it to Q and V projections in attention worked best per-parameter. Modern recipes often apply LoRA everywhere (including FFN) with a smaller rank.
4. Hyperparameters: rank r and scaling α
The forward pass is W₀x + (α/r)·BAx. The α/r scaling decouples 'how much capacity' (r) from 'how much effect' (α/r). Typical values: r=8, α=16 (so effective scale = 2). For high-data fine-tunes, bump r and α proportionally.
5. Merging vs Hot-Swapping at Inference
You can either merge BA into W₀ for zero-overhead inference (one model, one fine-tune), or keep them separate and hot-swap multiple LoRA adapters per request (the S-LoRA / Punica approach). The second mode is how multi-tenant LLM platforms serve thousands of customer fine-tunes from one base model.
Results
- ✓GPT-3 175B with LoRA: 0.01% of trainable params (~3.5M instead of 175B) — and equal quality
- ✓RoBERTa-Large: matches full FT on GLUE with 0.5% of trainable params
- ✓DeBERTa-XXL: matches full FT on SuperGLUE
- ✓No inference latency penalty after merging — the architecture stays the same shape
- ✓Spawned QLoRA (4-bit base + LoRA), enabling 70B fine-tune on a single 24GB consumer GPU
Takeaways
- →Low intrinsic rank is real. The 'update space' of a fine-tune is much smaller than the weight space.
- →Modularity wins. Adapters compose: stack multiple LoRAs for multi-skill models.
- →Serving economics changed. One base + N tiny adapters = N models for the price of 1 + ε.
- →Most fine-tunes don't need full weights. Default to LoRA/QLoRA unless you've proven you need more.