QKV Projection

Attention by shapes · 1 of 11

The idea

Why three maps out of one X? Because matching and delivering are different jobs. A query asks “what am I looking for?”; a key answers “what do I offer to be matched on?”; a value carries what actually gets handed over when you are picked. The match signal and the payload are allowed to be about different things.

The shapes

Drag the dimensions. Q and K must share a key dimension so KᵀQ is legal. V is free — it sets the width of what comes out. Model size is the row count of X; key and value dims are usually smaller, so the projections compress.

Live diagram

Next: Attention Computation