Fine-Tune · ICLR 2022
LoRA: Low-Rank Adaptation of Large Language Models
In short
Freeze the big model. Add tiny low-rank matrices to each weight. Train only those. Get ~10,000× fewer trainable params, same quality as full fine-tuning, and the ability to swap thousands of fine-tunes from one base model.
Why it matters
Before LoRA, fine-tuning a 70B model needed an enterprise-scale cluster. After LoRA, it fits on a single 24GB consumer GPU (with QLoRA). The entire 'custom LLM' industry. Every fine-tune marketplace, every domain-specific assistant, every multi-tenant LLM serving platform (S-LoRA, Punica, Lorax): exists because of this paper.
The key idea
Weight updates during fine-tuning have low intrinsic rank. So instead of updating the full W₀ (which is huge), parameterize ΔW = B·A where B is d×r and A is r×k for a tiny rank r (often 4-16). Train only B and A. Reconstruct W = W₀ + BA at inference.
How it works
1. The Math
For a frozen weight matrix W₀ ∈ ℝ^(d×k), define the fine-tuned weight as W = W₀ + ΔW where ΔW = BA, with B ∈ ℝ^(d×r) and A ∈ ℝ^(r×k). The rank r is tiny, often 4, 8, or 16. Total trainable params: r·(d+k) instead of d·k. For r=8, d=k=4096, that's a 256× reduction.
2. Initialization Trick
Initialize A with small random values and B with zeros. Then BA = 0 at the start, so the model's initial output exactly matches the base model. No warmup needed. The model 'learns the adapter from zero' without first having to undo a random perturbation.
3. Where to Apply It
The paper studied applying LoRA to Q, K, V, output, and the FFN. Result: applying it to Q and V projections in attention worked best per-parameter. Modern recipes often apply LoRA everywhere (including FFN) with a smaller rank.
4. Hyperparameters: rank r and scaling α
The forward pass is W₀x + (α/r)·BAx. The α/r scaling decouples 'how much capacity' (r) from 'how much effect' (α/r). Typical values: r=8, α=16 (so effective scale = 2). For high-data fine-tunes, bump r and α proportionally.
5. Merging vs Hot-Swapping at Inference
You can either merge BA into W₀ for zero-overhead inference (one model, one fine-tune), or keep them separate and hot-swap multiple LoRA adapters per request (the S-LoRA / Punica approach). The second mode is how multi-tenant LLM platforms serve thousands of customer fine-tunes from one base model.
Results
- GPT-3 175B with LoRA: 0.01% of trainable params (~3.5M instead of 175B), and equal quality
- RoBERTa-Large: matches full FT on GLUE with 0.5% of trainable params
- DeBERTa-XXL: matches full FT on SuperGLUE
- No inference latency penalty after merging: the architecture stays the same shape
- Spawned QLoRA (4-bit base + LoRA), enabling 70B fine-tune on a single 24GB consumer GPU
Takeaways
- Low intrinsic rank is real. The 'update space' of a fine-tune is much smaller than the weight space.
- Modularity wins. Adapters compose: stack multiple LoRAs for multi-skill models.
- Serving economics changed. One base + N tiny adapters = N models for the price of 1 + ε.
- Most fine-tunes don't need full weights. Default to LoRA/QLoRA unless you've proven you need more.
Keep learning
- Generative AI · Fine-Tuning & AlignmentLoRA, QLoRA, RLHF, DPO
- Deep Learning · Transfer & Modern ArchsTransfer learning, fine-tuning techniques
- Software Engineering for AI · GPU & Networking at ScaleLoRA Serving at Scale (S-LoRA, Punica)
- Linear Algebra · DecompositionsWhy low-rank matters (SVD intuition)