Foundations

Before anything else, text has to become numbers a model can do arithmetic on - and those numbers have to be trained toward something. This track is short on purpose: six ideas, each one load-bearing for everything after it.

The path: a token is chopped out of text, then embedded into a long vector. Two things happen to that vector next, independently - it gets directions it can be measured against (the dot product), and its raw scores get turned into a probability distribution (softmax). Context nudges the vector around as the sentence unfolds. All of it points at one objective: predict the next token. That single loss is what every later track - attention, efficiency, training, deployment - exists to serve better.

1. Tokens - text is chopped into pieces; a token alone carries no meaning.
2. Embeddings - each token becomes a long list of numbers, a point in space.
3. Directions & dot product - directions in that space can carry meaning; the dot product measures how much.
4. Context moves vectors - a word’s vector isn’t fixed; context pushes it around, one clue at a time.
5. Softmax & temperature - scores become a probability distribution; temperature controls how sharp it is.
6. The next-token objective - the one task every model trains on, at every scale.

Next: Attention & memory, where context stops being a nudge and becomes a mechanism.