Self vs Cross

Attention by shapes · 5 of 11

The idea

Put the last two pages side by side. Both ask with the same X. Self draws K and V from X → S is square. Cross draws them from E → S is rectangular. That single change is the whole difference.

What stays the same

Q and K still share a key dimension. V is still free. The rest of the pipeline — softmax, blend — does not care which sequence supplied the keys.

Live diagram

Next: Shared KV