GPT-2: the recipe
Training · 2019
The idea
Section titled “The idea”Everything abstract about pretraining becomes concrete in one run. Train a 2019-scale GPT end to end and you can watch the loss fall and the samples sharpen - from noise, to word-shaped noise, to prose (Karpathy demos a live run at ~0:31-0:42).
The recipe has barely changed since GPT-2: same objective, same architecture family, more data and compute. What changed is the price - a run that was a research-lab event in 2019 reproduces today for a few hundred dollars on rented GPUs. The lineage of this site’s film starts at exactly this model.
The paper’s own headline claim is bigger than the recipe: a 1.5B model doing 7-of-8 benchmarks in a zero-shot setting, no fine-tuning at all, trained on a new scraped corpus called WebText. This page uses GPT-2 as a cheap, concrete architecture to train live - not a re-run of the zero-shot result above.
The story, animated
Section titled “The story, animated”The recipe barely changed - watch the loss fall, the samples sharpen, and the price collapse.
Where it sits
Section titled “Where it sits”Worth having first: Pretraining
Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025), ch. 0:31-0:42.