TL;DR
A 671B-parameter Mixture of Experts model (only 37B active per token) trained for ~$5.5M to match GPT-4 quality — via Multi-head Latent Attention, FP8 training, and Multi-Token Prediction. Open weights.
Why it matters
DeepSeek-V3 (and the R1 model trained on top of it) was the 'DeepSeek moment' in early 2025 — proof that open-source models could match frontier closed models at a fraction of the cost. It validated MoE as the path forward for cost-efficient scale and shocked NVIDIA's stock price by hinting that compute requirements might not grow linearly with capability.
The Key Idea
Combine three engineering bets — sparse Mixture-of-Experts, Multi-head Latent Attention for KV cache compression, and FP8 mixed-precision training — to dramatically cut both training and inference cost while maintaining frontier quality.
How it works
1. Mixture of Experts (MoE) — 256 + 1 Shared
Each token routes through 1 shared expert (always active) + 8 of 256 routed experts. Total params: 671B. Active per token: only 37B. This is the cost-efficient scaling trick — you get a big model's capacity at a small model's inference cost. Auxiliary-loss-free load balancing keeps routing fair.
2. Multi-Head Latent Attention (MLA)
Standard attention's KV cache grows linearly with context length and number of heads — a memory disaster at scale. MLA compresses the KV cache by ~93% by projecting K and V into a small shared latent space, then decompressing per-head. Same quality, fraction of memory.
3. Multi-Token Prediction (MTP)
During training, predict the next 2 tokens in parallel instead of just one. Acts as a denser training signal. At inference time, the second predicted token can be used for speculative decoding, giving ~1.8× throughput speedup.
4. FP8 Mixed-Precision Training
Train in 8-bit floating point instead of bf16/fp16. ~2× memory reduction, faster matmuls, custom kernels for stability. Combined with careful loss scaling and tile-wise scaling for sensitive ops, they avoided the precision-loss disasters that have plagued prior FP8 attempts.
5. Pipeline & Communication Engineering
DualPipe scheduling overlaps computation with cross-node MoE communication so the network never blocks the GPUs. Trained on 14.8T tokens across 2,048 H800 GPUs, total cost ~$5.576M — a fraction of what GPT-4 reportedly cost (~$100M+).
Results
- ✓MMLU: 88.5 — matches or beats GPT-4o on knowledge benchmarks
- ✓HumanEval: 82.6 (Pass@1) — frontier coding
- ✓AIME 2024: 39.2 — best open-weights math reasoning at release
- ✓Total training cost: ~$5.576M — ≈30× cheaper than estimated GPT-4 training
- ✓Generation throughput: ~60 tokens/sec on a single inference node thanks to MTP + MLA
Takeaways
- →MoE is the path to cost-efficient scale at inference. Dense scaling is dead for frontier models.
- →KV cache is the new bottleneck at long context — compressing it (MLA) is huge.
- →Engineering tricks (FP8, MTP, DualPipe) compound. Each contributes ~1.3-2× somewhere.
- →Open-weights frontier exists. The gap between open and closed is closing fast.