All paper stories

MoE · arXiv

DeepSeek-V3 Technical Report

DeepSeek-AI, 2024. Read the paper on arXiv (opens a new tab)

In short

A 671B-parameter Mixture of Experts model (only 37B active per token) trained for ~$5.5M to match GPT-4 quality, via Multi-head Latent Attention, FP8 training, and Multi-Token Prediction. Open weights.

Why it matters

DeepSeek-V3 (and the R1 model trained on top of it) was the 'DeepSeek moment' in early 2025: proof that open-source models could match frontier closed models at a fraction of the cost. It validated MoE as the path forward for cost-efficient scale and shocked NVIDIA's stock price by hinting that compute requirements might not grow linearly with capability.

The key idea

Combine three engineering bets: sparse Mixture-of-Experts, Multi-head Latent Attention for KV cache compression, and FP8 mixed-precision training, to dramatically cut both training and inference cost while maintaining frontier quality.

How it works

  1. 1. Mixture of Experts (MoE): 256 + 1 Shared

    Each token routes through 1 shared expert (always active) + 8 of 256 routed experts. Total params: 671B. Active per token: only 37B. This is the cost-efficient scaling trick. You get a big model's capacity at a small model's inference cost. Auxiliary-loss-free load balancing keeps routing fair.

  2. 2. Multi-Head Latent Attention (MLA)

    Standard attention's KV cache grows linearly with context length and number of heads: a memory disaster at scale. MLA compresses the KV cache by ~93% by projecting K and V into a small shared latent space, then decompressing per-head. Same quality, fraction of memory.

  3. 3. Multi-Token Prediction (MTP)

    During training, predict the next 2 tokens in parallel instead of just one. Acts as a denser training signal. At inference time, the second predicted token can be used for speculative decoding, giving ~1.8× throughput speedup.

  4. 4. FP8 Mixed-Precision Training

    Train in 8-bit floating point instead of bf16/fp16. ~2× memory reduction, faster matmuls, custom kernels for stability. Combined with careful loss scaling and tile-wise scaling for sensitive ops, they avoided the precision-loss disasters that have plagued prior FP8 attempts.

  5. 5. Pipeline & Communication Engineering

    DualPipe scheduling overlaps computation with cross-node MoE communication so the network never blocks the GPUs. Trained on 14.8T tokens across 2,048 H800 GPUs, total cost ~$5.576M: a fraction of what GPT-4 reportedly cost (~$100M+).

Results

  • MMLU: 88.5: matches or beats GPT-4o on knowledge benchmarks
  • HumanEval: 82.6 (Pass@1): frontier coding
  • AIME 2024: 39.2: best open-weights math reasoning at release
  • Total training cost: ~$5.576M: ≈30× cheaper than estimated GPT-4 training
  • Generation throughput: ~60 tokens/sec on a single inference node thanks to MTP + MLA

Takeaways

  • MoE is the path to cost-efficient scale at inference. Dense scaling is dead for frontier models.
  • KV cache is the new bottleneck at long context: compressing it (MLA) is huge.
  • Engineering tricks (FP8, MTP, DualPipe) compound. Each contributes ~1.3-2× somewhere.
  • Open-weights frontier exists. The gap between open and closed is closing fast.

Read another paper