Attention is All You Need
TL;DR
Replace recurrence with self-attention. Train in parallel, get better quality, unlock every modern LLM, ViT, and multimodal model that followed.
Why it matters
This paper IS modern AI. Every GPT, Claude, Llama, Gemini, ViT, Whisper, Stable Diffusion text encoder, and multimodal model is built on the Transformer block. Before 2017 we trained sequence models with RNN/LSTM — slow, hard to parallelize, with vanishing gradients on long sequences. After 2017, we had a fully parallel architecture that just kept getting better with scale. Everything since is a remix.
The Key Idea
Self-attention lets every token attend to every other token in O(1) sequential depth — fully parallelizable on a GPU — unlike RNNs which require O(n) sequential steps and pass information through a tiny hidden-state bottleneck.
How it works
1. Scaled Dot-Product Attention
For each token, compute a Query (Q), Key (K), and Value (V) vector via three learned linear projections. The attention score between token i and token j is (Qᵢ · Kⱼ) / √d_k. Apply softmax across j to get weights, then take the weighted sum of V vectors. The √d_k normalization keeps the softmax from saturating at large dimensions.
2. Multi-Head Attention
Instead of one big attention, project Q/K/V into h smaller subspaces (heads), do attention in each, concatenate, and project back. Different heads learn to attend to different kinds of relationships — syntactic dependencies, coreference, positional patterns. The paper used h=8 heads.
3. Positional Encoding
Self-attention is permutation-invariant — it has no idea what order words come in. Fix this by adding a sinusoidal positional encoding to each input embedding, with different frequencies per dimension. The model learns to use these to recover word order. Modern variants (RoPE, ALiBi) improved on this idea.
4. Encoder-Decoder Stack
Stack N=6 encoder blocks (self-attention + FFN, with residual + LayerNorm) and N=6 decoder blocks (masked self-attention + cross-attention + FFN). The decoder's masked self-attention prevents future tokens from leaking back. This encoder-decoder split was original; modern LLMs (GPT) use decoder-only.
5. Position-wise Feed-Forward Network
After attention, each token independently passes through a 2-layer MLP (d_model → 4·d_model → d_model) with a ReLU/GELU in between. This is where most of the parameters live and most of the computation happens. In modern LLMs this is often the SwiGLU variant.
Results
- ✓WMT-14 EN→DE: 28.4 BLEU — new state of the art
- ✓WMT-14 EN→FR: 41.0 BLEU — state of the art at a fraction of the compute
- ✓Trained in 12 hours on 8 P100 GPUs (the base model); 3.5 days for the big model
- ✓Beat much larger ensembles of RNN-based models — parallelization was the unlock
Takeaways
- →Parallelization wins. RNNs were the bottleneck, not the model size.
- →Self-attention is enough. You don't need recurrence for long-range dependencies.
- →Positional info can be added separately from the embeddings — clean separation of concerns.
- →Scale is the way forward. The architecture left room for 1000× more parameters.