FlashAttention
The efficiency lineage · 2022
The idea
Section titled “The idea”Not an approximation - a rearrangement. It made long context affordable enough that the memory wall became the binding constraint.
A sequel, FlashAttention-2 (2023, arXiv:2307.08691), roughly doubled this again with better GPU work-partitioning - same idea, no new algorithm, just less time wasted on the chip.
Verdict
Section titled “Verdict”What it fixedExact attention computed without ever materialising the N x N matrix - IO-aware tiling.
What it still costsCompute is still quadratic and the KV cache still grows; it moves the wall, it does not remove it.
The story, animated
Section titled “The story, animated”Same bill, cheaper trip - the grid never touches slow memory, tile by tile.
Where it sits
Section titled “Where it sits”Worth having first: The memory wall
Unlocks: The machine underneath
Episode 4 · The Machine Underneath