Skip to content
AI, Explained

Gated DeltaNet

The efficiency lineage · 2024

The idea

Section titled “The idea”

The slot is clean — why does the box still fill up?

Delta rule can overwrite a fact when a new key hits the same slot. It cannot dim a fact that nobody asks about again. Old associations sit at full strength until a rewrite happens to find them.

Gated DeltaNet adds a forget-gate. One number, alpha, fades the whole state together — a volume knob on the box, not a rewrite of one slot. It keeps DeltaNet’s other dial, beta, for how hard this token writes. The gated step is then: read what the slot already says, erase with alpha, write only the leftover error, scaled by beta.

The leftover is the uniformity: every channel fades at the same rate. That is the next page.

Verdict

Section titled “Verdict”
What it fixedA learned gate lets the state forget.
What it still costsOne scalar rate per head - every feature channel fades together.

The story, animated

Section titled “The story, animated”

Read the slot, fade the box, write the leftover error. Then the cost of one knob.

Watch

Section titled “Watch”
Episode 2, beat 7 - the family · 124s, from the film series.
From the full film - the alpha dial · 97s, from the film series.

Where it sits

Section titled “Where it sits”

Worth having first: Delta rule

Unlocks: KDA

Episode 2 · The Linear Bet