Columns write back

Inside the blocks

The idea

Each column of the down-projection is a direction in embedding space. The firing neuron scales its column - basketball - and adds it to the stream. The fact lives in the weights.

Try it

Where it sits

Worth having first: Rows ask questions

Unlocks: Mixture of Experts · What it can’t remember

Chapter: how facts are stored (parts bin D8)