Fused QKV

Attention by shapes · 8 of 11

The idea

In code, the three projections per head are often one fat matmul: Wqkv has height H×(dk+dk+dv), then the result is sliced into the per-head triples. Same math as separate weights; one trip through the GEMM.

The shapes

Drag Heads — the fused weight’s row count tracks H×(dk+dk+dv).

Live diagram

Next: Single vs Multi-Head