Backpropagation
Training
The idea
Section titled “The idea”Training is measurement plus blame: run the model, measure how wrong the next-token guess was, then nudge every one of the billions of dials a hair in the direction that would have made it less wrong. Repeat.
The demystifying move: shrink the network until you can read it. A toy with six parameters is just ordinary arithmetic - multiplies, adds, an exponent, a division - composed into one expression; blame flows backwards through those operations by calculus, nothing more exotic (ch. 0:20-0:22). A real model is the same picture with billions of dials, which changes the engineering, not the idea.
The story, animated
Section titled “The story, animated”Six dials, ordinary arithmetic. Measure how wrong. Blame flows backwards — the same 3% to 4% nudge, from the other side.
Where it sits
Section titled “Where it sits”Worth having first: The next-token objective
Unlocks: Pretraining
Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).