The training data
Training
The idea
Section titled “The idea”Before anything trains, the internet is funnelled. A raw crawl is mostly markup, boilerplate, spam and duplicates; pipeline stages strip the HTML, filter by language and quality, remove personal data and dedupe - and terabytes of pages reduce to a comparatively small library of clean text, measured in trillions of tokens (Karpathy walks a FineWeb-style funnel at ~0:01-0:07).
The point that changes intuitions: the dataset is a curated artifact someone built, with judgement calls at every filter. What a model can know starts here, before a single weight updates.
The story, animated
Section titled “The story, animated”Five gates, one pass - what a raw crawl looks like before and after judgment calls.
Where it sits
Section titled “Where it sits”Worth having first: Tokens
Unlocks: Pretraining
Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025), ch. 0:01-0:07.