Predict the gist, not the pixels
World models · 2026 · Part 5 of a series that starts with The drone and the world model
Drawing every pixel of the future is wasteful. This page follows the move to predicting compact codes instead, and the trap that comes with it.
World models · part 5
Predict the Gist, Not the Pixels
A video world model redraws every leaf and reflection before it can say what happens next. What if it only predicted the parts of the future that matter? That idea is JEPA. It saves enormous effort, and it hides a trap: a model that predicts nothing at all scores perfectly.
This page starts with the oldest way to store “what works where”, the Q-table, then moves into latent space, shows the collapse trap in a live lab, and ends with the same idea applied to language models. Wherever a shortcut version is commonly said or written, it's marked COMMON SLIP and the correct form is shown.
1 · The table that doesn't scale
Q-learning keeps one number per state and move
Classic Q-learning keeps a table with one row per state and one column per action, |S| × |A| cells in all. Each cell holds Q(s,a), the agent's estimate of the total future reward for taking action a in state s and playing well afterwards. Every time it tries a cell, it nudges that estimate:
The trouble is the size of the table. Every cell has to be visited several times before its number means anything. Squeezing states together with a feature map f(s), so the table becomes Q(f(s), a), shrinks it, at the price of treating different states as the same.
Lab 1 · Fill the table
an agent wanders an 8×8 grid at random; each cell lights up as it is triedCells tried at least 5 times
64 squares × 4 moves is a toy. Part 2 counted 1044 chess positions and 10170 Go positions. Deep RL replaced the table with a network that maps a state to its Q-values, which is f(s) learned instead of chosen by hand.
2 · Pixels or latents
Predict the code, not the picture
A pixel world model learns T̂(st+1 | st, at) where every s is a full image. Most of its effort goes on detail no decision depends on: the exact pattern of leaves, glare, sensor noise. Some of that detail is unpredictable even in principle, so the model is punished for things it could never know.
A JEPA (joint-embedding predictive architecture) first passes each frame through an encoder with weights θ, giving a short code z. A small predictor guesses the next code from the current one, and training compares codes, never pixels:
3 · The trap
If every code is zero, the prediction is perfect
The encoder is trained by the same loss. Its laziest solution is to output the same vector for every frame. Then the predictor just repeats it, the error is zero, and the codes say nothing about the world. This is representation collapse.
The cure is a second term that refuses to let the codes bunch up. VICReg (2021) pushes the spread of each code dimension up to at least 1 and keeps the dimensions uncorrelated. SIGReg, from LeJEPA (2025), checks the batch of codes along many random directions and pushes it towards a round, standard Gaussian, z ~ N(0, I). LeWorldModel (2026) uses SIGReg to train a JEPA world model straight from pixels.
Lab 2 · Watch it collapse
64 frames of a dot on a circle; the next frame turns about 0.4 rad, give or take; codes are 2-DNumbers
Each frame is the dot's position plus a third, pure-noise number. The encoder is a 2×3 matrix and the predictor a 2×2 matrix, trained together by gradient descent on the loss you choose. The turn to the next frame varies randomly (spread 0.35 rad), so no code can predict it exactly; shrinking the codes shrinks that unavoidable error, which is what drives the collapse. The spread penalty is VICReg-style: (1 − std)² for each code dimension below 1, plus the squared correlation between the two dimensions, weighted by 5.
Regularisers like these are one family of cures. Another keeps the target side out of training: the target codes come from a slowly updated copy of the encoder and get no gradient, as in I-JEPA and V-JEPA. The site's JEPA, LeJEPA and LeWorldModel pages go deeper.
4 · The same idea for language
Predict the next word's vector, not its probability
A language model reads token ids, looks each up in an embedding table E, and at the end turns its final hidden state h into a probability for every word in the vocabulary. Training compares that list with the true next token (cross-entropy).
The ids are made up for illustration.
The JEPA-flavoured idea: skip the vocabulary step during training and ask the hidden state to land on the next token's embedding directly.
Why the pure swap fails
‖h − e‖2 on its own walks straight into section 3's trap. The embedding table E is learned too, so the cheapest answer is to make every embedding the same vector; then any h that matches it scores zero. You would also have no way to pick a word at the end. Published work keeps the next-token loss and adds an embedding-space term on top: LLM-JEPA reports better fine-tuning results that way across several model families.
5 · Next
What world models still get wrong
Latent or pixel, today's world models share a list of open problems: rare events, limited precision, slow adaptation and missing senses. The last page in this series goes through them.
Based on minutes 57–63 of the YC Decoded video “World Models: An intuitive introduction” (July 2026). The equations were checked against the video's whiteboard and corrected where needed.
Sources. Q-learning · VICReg (2021) · I-JEPA (2023) · LeJEPA and SIGReg (2025) · LeWorldModel (2026) · LLM-JEPA (2025)