Architecture · 8 min read · Updated 2026-05
Transformer Architecture Explained (2026)
The Transformer is the architecture behind GPT, Claude, Llama, Gemini, ViT, and Stable Diffusion's text encoder. It replaced recurrence with self-attention in 2017 and has stayed the dominant design ever since. Here's how it actually works.
Why it beat RNNs
RNNs process tokens one at a time: slow on GPUs, hard to scale. Transformers process all tokens in parallel via self-attention. Combined with residual connections and layer normalization, this made it possible to stack much deeper models and train them faster.
Self-attention in one paragraph
For every token, learn three projections: Query (Q), Key (K), Value (V). Compute attention scores as (Q · Kᵀ) / √d_k. Apply softmax to get weights. Sum the V vectors weighted by those scores. The result for token i is a mix of all tokens' V vectors, weighted by how relevant they are to token i.
Multi-head attention
Instead of one big attention, split Q/K/V into h smaller subspaces (heads) and do attention in each. Concatenate the outputs and project back. Different heads learn different relationship patterns: syntactic dependencies, coreference, positional patterns.
Positional encoding
Self-attention has no idea of word order. Fix this by adding a position vector to each input embedding. Original paper used sinusoidal encodings; modern models use RoPE (Rotary Positional Embeddings, in Llama) or ALiBi (which extrapolates to longer sequences).
Encoder, decoder, or both
Three flavors of Transformer:
- Encoder-only (BERT): bidirectional context, masked language modeling, great for classification/embedding
- Decoder-only (GPT, Llama, Claude): causal masking, next-token prediction, the chat-model family
- Encoder-decoder (T5, original 2017 paper): separate encoder reads input, decoder generates output
What changed in 2025-2026
The core block is unchanged, but production transformers have evolved:
- Attention variants: MHA → MQA → GQA (Llama 3) for cheaper inference
- FlashAttention v3: memory-efficient attention; required for long context
- RoPE positional encoding: better extrapolation than sinusoidal
- Mixture of Experts (MoE): sparse activation in the FFN; DeepSeek-V3 popularized it
- Multi-head Latent Attention (MLA): compresses KV cache, used in DeepSeek-V3
Frequently asked questions
Is the Transformer still the best architecture?
For LLMs, yes, though State-Space Models (Mamba, Mamba-2) are a credible alternative for very long context. In 2026, all frontier models are Transformer-based with various optimizations.
How is a 'decoder-only' model different from 'encoder-decoder'?
Decoder-only models use a single causal-masked attention stack. Every token only attends to past tokens. Encoder-decoder uses two stacks: a bidirectional encoder reads the full input, then a decoder generates output while cross-attending to the encoder.
Why √d_k in the attention formula?
Without scaling, dot products grow with dimension and push softmax into saturation (gradients near zero). Dividing by √d_k keeps the variance of the dot products at ~1 regardless of dimension.