Mixture of Experts

Inside the blocks · 2021+

The idea

Most of the parameters live in the MLPs - so keep many and route.

PaperSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient SparsityFedus, Zoph, Shazeer · Google · 2021arXiv:2101.03961 →

Sparse MoE itself is older (Shazeer et al., 2017) - Switch Transformer is the paper that made it practical at scale: top-1 (not top-2) routing, and the first trillion-parameter model actually trained. DeepSeekMoE (2024, arXiv:2401.06066) is the more recent refinement - many small, fine-grained experts plus a few always-on shared ones - and is the specific recipe behind Kimi K3’s own expert layout, though this project’s film never names it as such.

The mechanics, in one pass: a small router scores each token against every expert and keeps the top-k. The experts’ weights live stacked in one tensor; tokens are sorted by their assigned expert, each expert multiplies its contiguous slice, and results scatter back to token order. That sort-then- group step is not decoration - it is what makes MoE fast on real hardware (see Batching & GPU kernels).

Kimi K3's real layout: 898 experts per layer — 2 shared ones fire on every token (grammar, structure), 16 of the 896 routed ones are picked per token, the rest draw zero compute. Numbers from the film's own research (act 3–4).

Verdict

What it fixedParameter count grows without growing per-token compute - each token visits a few expert MLPs, not one giant one.

What it still costsAll experts must sit in memory, and routing adds a failure surface of its own.

The story, animated

Every token visits a few experts, not the whole network. Watch the router pick, then the tokens sort into groups a GPU can multiply.

Watch

From the full film - Mixture of Experts, sparse capacity · 110s, from the film series.

Where it sits

Worth having first: Columns write back