The efficiency lineage

The memory wall sets the problem: the KV cache grows with every token of context, and eventually memory - not compute - is what stops you. Every page in this track is someone’s answer to that one problem, and they split into a few genuinely different strategies.

Shrink what you keep. MQA shares one key/value across all heads; GQA walks that back to groups of heads for more quality; MLA compresses what’s kept into one small latent. All three still grow with context - they just grow slower.

Don’t keep it at all. Linear attention replaces the growing cache with a fixed-size state. That state can’t do exact retrieval, so the rest of the branch is a sequence of repairs: the delta rule overwrites by error instead of blindly accumulating, Gated DeltaNet adds a forgetting gate, KDA makes that gate per-channel instead of uniform.

Or just change the mask. Linearizing papers compared against a window that drops the first tokens. Keep those tokens (attention sinks) and sliding-window attention — zero extra training — matches or beats the conversion on short benches. The leftover: a needle still has to land in the window. This is a sibling of the linear branch, not a step toward the hybrid.

Solve a different problem. FlashAttention and sparse attention don’t touch memory at all - they make the same growing cache cheaper or faster to compute over. Useful, but orthogonal to everything above.

The two memory-side branches don’t stay parallel forever - they converge. SWA is a sibling of Linear, not part of that merge.

Every page in this track answers the same question the memory wallraises: keep the cache, shrink it, or replace it? Two branches - shrink-what-you-keep and don't-keep-it-at-all - run in parallel and converge: the hybrid is KDA and MLA composed, and it is the architecture the rest of this site's film is actually about.Sliding window with sinks is a sibling of Linear, not a repair of it — a training-free mask compared against the conversion. FlashAttention and sparse attention solve a different problem - compute, not memory - so they sit outside the convergence.

1. MQA - one K/V shared by all heads; the cache shrinks by the head count.
2. GQA - groups of heads share K/V; most of the saving, most of the quality back.
3. MLA - K and V compressed into one small latent; a large cache cut at matched quality.
4. FlashAttention - exact attention, computed without ever materialising the N x N matrix.
5. Sparse attention - attend only to the top-k relevant tokens.
6. Linear attention - the cache becomes one fixed-size state that never grows.
7. Sliding window with sinks - a training-free mask that matches or beats linear retrofits; the needle still has to land in the window.
8. Delta rule - overwrite by error: recall what the state says, write only the difference.
9. Gated DeltaNet - a learned gate lets the state forget.
10. KDA - a gate per channel; every row of the state gets its own lifetime.
11. The hybrid - KDA and MLA composed, three-to-one - the architecture the film is about.

Next: Inside the blocks, where the other half of a transformer block - the MLP - does its own work.