Language Models are Few-Shot Learners (GPT-3)
TL;DR
Scale a Transformer to 175B parameters, train on 300B tokens. Suddenly it solves new tasks from a few prompt examples alone — no fine-tuning needed. Birth of prompt engineering and the direct ancestor of ChatGPT.
Why it matters
This paper made in-context learning real and observable at scale. Before GPT-3 you fine-tuned a model per task. After GPT-3 you wrote a prompt. It triggered the entire prompt-engineering field, validated the 'scale is what matters' hypothesis, and led directly to InstructGPT → ChatGPT → the modern LLM industry.
The Key Idea
At enough scale, language models become few-shot learners — they can adapt to new tasks from prompt examples alone, with no gradient updates. The 'API' becomes natural language.
How it works
1. Same Architecture, 100× Bigger
GPT-3 reused the GPT-2 decoder-only Transformer architecture — no new blocks, no fancy tricks. They just scaled it: 96 layers, 96 attention heads, d_model = 12,288. The largest of 8 model sizes, from 125M to 175B parameters.
2. Training Data: 300B Tokens
Mixture of Common Crawl (filtered for quality), WebText2 (curated Reddit links), Books1 + Books2, and Wikipedia — weighted to upsample high-quality sources. Critical insight: data quality matters as much as quantity.
3. The Few-Shot Evaluation Paradigm
They introduced three regimes: zero-shot (just task description), one-shot (one example in prompt), and few-shot (10-100 examples). No gradient updates between examples. This 'in-context learning' was the actual breakthrough — proof that pretraining alone was enough for many tasks.
4. Sparse Attention for Long Context
To handle 2048-token context efficiently, they used alternating dense and locally-banded sparse attention layers (from Sparse Transformers). This was later replaced by Flash Attention in modern models.
Results
- ✓LAMBADA (next-word prediction): 86.4% — beat fine-tuned SOTA in zero-shot
- ✓TriviaQA: 71.2% in few-shot — outperformed fine-tuned T5-11B
- ✓SuperGLUE: near-human in few-shot setting
- ✓Performed news article writing humans couldn't reliably distinguish from real ones
- ✓Cost (estimated): a few million dollars to train — at the time, unprecedented
Takeaways
- →Scale unlocks emergent abilities that weren't present at smaller sizes.
- →In-context learning works — pretrain once, deploy everywhere via prompts.
- →Data quality and mixture matter as much as parameter count.
- →The prompt is the new API. Prompt engineering became a real discipline overnight.