Single vs Multi-Head

Attention by shapes · 9 of 11

The idea

The structural fact worth holding: if total width is fixed (d = H×dv), multi-head does not change the V×A multiply count — it is still N²×d. It does pay H× more dot products for the scores. The win is several views, not a free lunch on arithmetic.

The shapes

Left single-head, right H heads. Watch the two ledgers as you drag N, d, H.

Live diagram

Next: MQA (shapes)