Tokens

Foundations

The idea

Text is chopped into tokens - words and word-pieces. Each is just an id in a vocabulary. Alone, a token carries no meaning at all; everything that follows is about giving it some.

The real design question behind tokenization is a trade: symbols versus sequence length. Raw bits give two symbols and absurdly long sequences; bytes give 256 and still too long; BPE keeps merging the most frequent pairs until the vocabulary hits a sweet spot around 100k entries - GPT-4’s tokenizer has exactly 100,277 (ch. 0:08-0:12). A 15-trillion-token training corpus is measured after this compression.

The compression has a price the rest of this site keeps meeting: the model never sees letters, only opaque chunks. “Ubiquitous” arrives as three fragments, which is why letter-counting and spelling are weirdly hard for systems that write sonnets (ch. 2:01-2:04) - see Jagged intelligence.

Try it

Try: type anything - then hit UPPERCASE it and watch the token count jump for text that looks the same to you.

loading the tokenizer…

Where it sits

Unlocks: Embeddings · The next-token objective

Source-checked against Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).