All paper stories

Scale · NeurIPS 2020

Language Models are Few-Shot Learners (GPT-3)

Tom B. Brown, Benjamin Mann, Nick Ryder, et al. (OpenAI), 2020. Read the paper on arXiv (opens a new tab)

In short

Scale a Transformer to 175B parameters, train on 300B tokens. Suddenly it solves new tasks from a few prompt examples alone. No fine-tuning needed. Birth of prompt engineering and the direct ancestor of ChatGPT.

Why it matters

This paper made in-context learning real and observable at scale. Before GPT-3 you fine-tuned a model per task. After GPT-3 you wrote a prompt. It triggered the entire prompt-engineering field, validated the 'scale is what matters' hypothesis, and led directly to InstructGPT → ChatGPT → the modern LLM industry.

The key idea

At enough scale, language models become few-shot learners. They can adapt to new tasks from prompt examples alone, with no gradient updates. The 'API' becomes natural language.

How it works

  1. 1. Same Architecture, 100× Bigger

    GPT-3 reused the GPT-2 decoder-only Transformer architecture. No new blocks, no fancy tricks. They just scaled it: 96 layers, 96 attention heads, d_model = 12,288. The largest of 8 model sizes, from 125M to 175B parameters.

  2. 2. Training Data: 300B Tokens

    Mixture of Common Crawl (filtered for quality), WebText2 (curated Reddit links), Books1 + Books2, and Wikipedia: weighted to upsample high-quality sources. Critical insight: data quality matters as much as quantity.

  3. 3. The Few-Shot Evaluation Paradigm

    They introduced three regimes: zero-shot (just task description), one-shot (one example in prompt), and few-shot (10-100 examples). No gradient updates between examples. This 'in-context learning' was the actual breakthrough: proof that pretraining alone was enough for many tasks.

  4. 4. Sparse Attention for Long Context

    To handle 2048-token context efficiently, they used alternating dense and locally-banded sparse attention layers (from Sparse Transformers). This was later replaced by Flash Attention in modern models.

Results

  • LAMBADA (next-word prediction): 86.4%: beat fine-tuned SOTA in zero-shot
  • TriviaQA: 71.2% in few-shot: outperformed fine-tuned T5-11B
  • SuperGLUE: near-human in few-shot setting
  • Performed news article writing humans couldn't reliably distinguish from real ones
  • Cost (estimated): a few million dollars to train, at the time, unprecedented

Takeaways

  • Scale unlocks emergent abilities that weren't present at smaller sizes.
  • In-context learning works: pretrain once, deploy everywhere via prompts.
  • Data quality and mixture matter as much as parameter count.
  • The prompt is the new API. Prompt engineering became a real discipline overnight.

Read another paper