Watch first, then practise
World models · 2026 · Part 4 of a series that starts with The drone and the world model
Labelled robot data is scarce and video is nearly free. This page shows how far watching gets a world model, and how few labels it needs afterwards.
World models · part 4
Watch First, Then Practise
Labelled robot data is slow and expensive. Video of the world is nearly free. Can a machine learn how the world works by watching, then learn to act with only a little hands-on practice?
This page builds the recipe today's world models follow, shows why they look at more than one frame, and traces the idea from 2018 to 2026. Wherever a shortcut version is commonly said or written, it's marked COMMON SLIP and the correct form is shown.
1 · The recipe
Watch a lot, practise a little, train in the dream
Today's world models follow three steps:
- Watch. Pre-train on lots of unlabelled video. The model learns how the world moves on its own: how things fall, slide, coast and bump.
- Practise a little. Fine-tune on a small action-labelled set, so the model learns what your actions do: Tψ(st+1 | st, at).
- Dream. Train or plan the policy πθ inside the model's imagined rollouts, then act in the real world.
Why should watching help, if video has no actions? Because most of what a world model must learn is how the world behaves without you. Only a small part is what your controls do. The lab makes that concrete.
Lab 1 · Get the cart to the flag
the cart must reach the flag at 6 m after 8 s, slowly; pushes are planned in the model, then played blindWhat the model learned
| part of the world | learned from | learned | true |
|---|
Typical miss over 30 fresh datasets
The world here is a cart on a gentle uphill track: dv/dt = g·a − μv − c·v|v| − s, where a is the push. “Video” is 3,000 noisy moments with no pushes, which pins down μ, c and s. The labelled examples are moments with a random push and its result, the only data that says anything about g. “Labels only” must learn all four numbers from the labels. The plan is a full push, then a full brake, then coasting, with timings chosen by searching inside the learned model.
This toy is kinder than reality: real pixels are not four numbers, and the real “passive” part is enormous. That is the point. The bigger the share of the world that behaves the same whatever you do, the more video can teach and the fewer expensive labels you need. Playing the plan blind also exaggerates the damage of a bad model. A real controller re-plans as it goes, as the drone did in part 1.
2 · How much past to show
One picture can't tell you which way things move
A single frame shows where things are, not how fast they go. A model fed one frame has to guess the motion; give it two and the difference between them reveals velocity. That's why many video world models condition on several past frames. A weather model built this way, GenCast, conditions each 12-hour step on the two previous states of the atmosphere, then draws the next state with a diffusion model.
Lab 2 · Predict the next frame
a ball moves at constant speed; dots are the futures the model must keep open3 · How we got here
Eight years from a dreaming car to robots
World Models. A small autoencoder squeezes game frames into a code; a recurrent network predicts the next code. In the car-racing game, the controller is trained in the real game on the model's features. In a Doom dodging game, it is trained entirely inside the model's dream and still works in the real game. The agent also learns to exploit flaws in its own dream, so the authors make the dream noisier.
PlaNet plans inside a learned latent model, directly from pixels.
Dreamer trains its policy on imagined rollouts in that latent space. DreamerV3 (Jan 2023) collects diamonds in Minecraft from scratch.
GAIA-1, a generative world model for driving video, from Wayve. The company raised a $1.05B Series C in May 2024.
GenCast, a diffusion weather model that beats the leading physics-based ensemble forecast on most of the targets it was tested on.
Dreamer 4 learns from gameplay video plus a small action-labelled slice, then trains its agent purely in imagination: the first agent to get Minecraft diamonds from offline data alone.
DreamZero, a “World Action Model”: a 14-billion-parameter video diffusion model (Wan2.1) extended to predict robot actions together with future video. It runs at about 7 actions per second and adapts to a new robot from about 30 minutes of play data.
Common slips in this story
- “World Models trained both games in the dream.” Only the Doom agent was; the racing controller learned in the real game.
- “Dreamer 1 came out in November 2018.” That was PlaNet. Dreamer followed in December 2019.
- “Wayve raised $1.5B.” Its May 2024 round was $1.05B.
- “DreamZero needed only 500 hours of teleoperation.” We couldn't trace that number. The paper's own headline is 30 minutes of play data to adapt to a new robot.
4 · The brain analogy, carefully
Does the cortex predict the result of your actions?
One 2024 theory of the neocortex, “active predictive coding”, says yes:
“each cortical area estimates both latent sensory states and actions … and the cortex as a whole predicts the consequences of actions at multiple hierarchical levels.”
That is the same split as model-based RL from the previous page: a policy that picks actions, and a model that predicts their consequences. It is one theory among several, not settled science.
Common slip: “fish and ants are pure reflex”
The tidy story says simple animals only map senses to moves, a “lizard brain”, and only mammals predict. Neuroscience abandoned the lizard-brain picture long ago. Insects and fish send a copy of each motor command to their sensory systems (an efference copy) to predict and cancel the feeling of their own movement. That is a small forward model, a T for one step ahead.
Common slip: “sleep replays the day in reverse at 7× speed”
During sleep, bursts in the hippocampus called sharp-wave ripples do replay sequences from the day, compressed roughly 5 to 20 times. Most sleep replay runs forward. Reverse replay was first found during pauses while awake. Replay is thought to help consolidate memory; that it trains a policy the way imagined rollouts train Dreamer is an appealing analogy, not a finding.
5 · Next
Predicting pixels is wasteful
Every model on this timeline that generates video spends most of its effort drawing pixels: leaves, reflections, noise that no decision depends on. The next page asks whether a world model could predict only the gist, and what goes wrong when it tries.
Based on minutes 41–57 of the YC Decoded video “World Models: An intuitive introduction” (July 2026). The equations were checked against the video's whiteboard and corrected where needed.
Sources. World Models (2018) · PlaNet (2018) · Dreamer (2019) · GAIA-1 (2023) · Wayve's 2024 round · GenCast (Nature 2024) · Dreamer 4 (2025) · World Action Models are Zero-shot Policies (2026) · A sensory–motor theory of the neocortex (2024) · The triune (“lizard”) brain model · Efference copy · Sharp waves and ripples