Rows ask questions

Inside the blocks

The idea

Inside an MLP, each row of the up-projection is a question asked by dot product, the bias sets the bar, and the ReLU gate kills every answer below it. Try the presets below - only the full name clears the bar.

However many questions the rows ask, they ask them in one pass - the block spends the same compute on a trivial token as on a pivotal one. When a problem needs more work than one pass holds, the only place to get it is more tokens: see Tokens to think (ch. 1:48).

A naming note for reading real code: modern MLPs usually come in threes, not twos - a gate projection and an up projection whose outputs multiply (a smooth gate like SiLU in place of our toy ReLU), then the down projection. Same shape of story - questions, a gate, an answer written back - one extra matrix.

Try it

Where it sits

Worth having first: Directions & dot product · The residual stream

Unlocks: Columns write back

Chapter: how facts are stored (parts bin D7)

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).