The memory wall

Attention & memory

The idea

Two costs grow with context length N: compute (every token compares with every earlier one - N x N) and memory (the cache holds 2 x N x d numbers per layer). At long context, memory - not math - is the wall. Everything downstream is an answer to this node.

And nothing survives the conversation: each session is a fresh instantiation that reads its own context and is discarded. Whatever isn’t in the weights or the current window does not exist for the model (ch. 1:41-1:42) - which is why the whole lineage below fights over the cost of that window.

The story, animated

The same argument, slide by slide - watch both bills grow, then see which one actually stalls the chip.

Where it sits

Worth having first: KV cache

Unlocks: MQA · FlashAttention · Sparse attention · Linear attention

Episode 1 · The Memory Problem

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).