Pretraining
Training
The idea
Section titled “The idea”Backprop at planetary scale: trillions of tokens, one objective. This is where the grammar, the facts and the geometry of the embedding space all get baked into the weights.
The scale, in checkable numbers: a modern open corpus like FineWeb is ~44 TB raw, ~15 trillion tokens once tokenised, distilled from a crawl indexing billions of pages (ch. 0:02-0:14; the funnel itself lives on The training data).
And the cost curve bends fast: GPT-2’s 2019 run cost on the order of $40k; a careful 2024 reproduction gets there in about a day for a few hundred dollars - better data, faster hardware, better software, all at once (ch. 0:33-0:34; the worked case is GPT-2: the recipe).
The story, animated
Section titled “The story, animated”One objective, repeated at planetary scale - then the same corpus and cost numbers, for real.
Where it sits
Section titled “Where it sits”Worth having first: The next-token objective · Backpropagation
Unlocks: Scaling laws · Instruction tuning & RLHF
Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).