T
TRKFLY AI
← All guides
MoE5 min read · Updated 2026-05
🧠

What is Mixture of Experts (MoE)?

Mixture of Experts is the architecture that lets a 671B-parameter model run at a 37B-parameter inference cost. Instead of every token using every weight, a router sends each token to only a few specialized 'expert' sub-networks. Sparse activation, dense capacity.

The intuition

Take the Feed-Forward Network (FFN) layer of a Transformer and split it into N independent FFNs (the experts). For each token, a small router network picks the top-k experts (typically k=2 or k=8). Only those experts compute; the rest stay idle.

Total params vs active params

A 'sparse 671B' model has 671B total params on disk but only ~37B active per token. You pay disk-and-VRAM cost for 671B but compute cost for 37B. That's the trick.

Why MoE is the path forward for scale

Dense scaling (just making the FFN bigger) has hit a wall: training cost scales quadratically in some regimes, and inference becomes expensive. MoE breaks that. You can grow total parameters (so the model knows more) without proportionally growing compute per token (so inference stays affordable).

  • →DeepSeek-V3 (2024) — 671B total, 37B active, frontier quality at a fraction of dense cost
  • →Mixtral 8×7B (2023) — 47B total, ~13B active, popularized MoE in open-weights
  • →Switch Transformer (2021) — Google's early MoE; trillion-param paper that started the modern wave

Engineering challenges

MoE is not free. The hard parts:

  • →Load balancing — without it, the router collapses to one favorite expert and the rest die
  • →Communication overhead — experts often live on different GPUs; all-to-all comms can dominate runtime
  • →Memory — you still need to hold all experts in VRAM somewhere
  • →Stability — MoE training can diverge in ways dense training doesn't

What changed with DeepSeek-V3

Three techniques DeepSeek combined to make MoE actually practical at frontier scale:

  • →Auxiliary-loss-free load balancing — bias terms on each expert keep routing balanced without the usual load-balancing loss
  • →Multi-head Latent Attention (MLA) — compresses KV cache by ~93%, solving the memory side
  • →FP8 training + DualPipe scheduling — overlaps compute with all-to-all comms

Frequently asked questions

▸Does every Transformer layer get MoE?

Usually only the FFN layers; attention layers stay dense. Some designs alternate MoE and dense FFNs every other layer.

▸Is MoE faster at inference than a dense model with the same active parameters?

About the same in compute, but MoE has higher memory and worse cache locality. Dense usually wins on a single GPU; MoE wins on multi-GPU setups where you can shard experts across devices.

▸Can I fine-tune an MoE model with LoRA?

Yes — apply LoRA to attention projections (Q/V) and optionally to the shared expert. Per-expert LoRA is an active research area but not standard yet.

Keep learning — roadmaps
🗺️
Generative AI · Modern LLM Architecture
🗺️
Software Engineering for AI · Inference at Scale
Read the papers
📄
DeepSeek-V3 (2024)
Browse all guides →