Scale · NeurIPS 2020
Language Models are Few-Shot Learners (GPT-3)
In short
Scale a Transformer to 175B parameters, train on 300B tokens. Suddenly it solves new tasks from a few prompt examples alone. No fine-tuning needed. Birth of prompt engineering and the direct ancestor of ChatGPT.
Why it matters
This paper made in-context learning real and observable at scale. Before GPT-3 you fine-tuned a model per task. After GPT-3 you wrote a prompt. It triggered the entire prompt-engineering field, validated the 'scale is what matters' hypothesis, and led directly to InstructGPT → ChatGPT → the modern LLM industry.
The key idea
At enough scale, language models become few-shot learners. They can adapt to new tasks from prompt examples alone, with no gradient updates. The 'API' becomes natural language.
How it works
1. Same Architecture, 100× Bigger
GPT-3 reused the GPT-2 decoder-only Transformer architecture. No new blocks, no fancy tricks. They just scaled it: 96 layers, 96 attention heads, d_model = 12,288. The largest of 8 model sizes, from 125M to 175B parameters.
2. Training Data: 300B Tokens
Mixture of Common Crawl (filtered for quality), WebText2 (curated Reddit links), Books1 + Books2, and Wikipedia: weighted to upsample high-quality sources. Critical insight: data quality matters as much as quantity.
3. The Few-Shot Evaluation Paradigm
They introduced three regimes: zero-shot (just task description), one-shot (one example in prompt), and few-shot (10-100 examples). No gradient updates between examples. This 'in-context learning' was the actual breakthrough: proof that pretraining alone was enough for many tasks.
4. Sparse Attention for Long Context
To handle 2048-token context efficiently, they used alternating dense and locally-banded sparse attention layers (from Sparse Transformers). This was later replaced by Flash Attention in modern models.
Results
- LAMBADA (next-word prediction): 86.4%: beat fine-tuned SOTA in zero-shot
- TriviaQA: 71.2% in few-shot: outperformed fine-tuned T5-11B
- SuperGLUE: near-human in few-shot setting
- Performed news article writing humans couldn't reliably distinguish from real ones
- Cost (estimated): a few million dollars to train, at the time, unprecedented
Takeaways
- Scale unlocks emergent abilities that weren't present at smaller sizes.
- In-context learning works: pretrain once, deploy everywhere via prompts.
- Data quality and mixture matter as much as parameter count.
- The prompt is the new API. Prompt engineering became a real discipline overnight.