World models vs RL
World models · 2026 · Side lesson to Too many moves, too few labels
World models and reinforcement learning are often pitched as rival approaches. This page sorts methods by two questions, whether they have a world model and whether they learn from reward, and shows with live labs what a model buys and what it costs.
World models · side lesson
World Models vs RL
“World models” and “reinforcement learning” are often talked about as rival roads to smarter machines. They aren't rivals. A world model is a part. You can bolt it onto reinforcement learning, use it with no reinforcement learning at all, or leave it out. The useful question is which parts an agent has, and where it spends its effort.
This page lays out the four combinations, shows in a live lab what a model buys you, shows what it costs, and checks the claims against published results.
1 · The map
Two questions sort every method
Ask two things about any agent. Does it have a world model, something that predicts T(st+1 | st, at)? And does it learn from reward, improving by trying things and being scored? Reinforcement learning is the “learns from reward” column. World models are the “has a model” row. They cross.
Lab 1 · Pick a corner
tap a cell; the loop below shows what flows whereCommon slip: “world models will replace RL”
The strongest world-model agents are reinforcement learners. Dreamer, MuZero and TD-MPC2 all learn from reward; the model changes how they learn, not whether. The real contrast is model-free versus model-based, and separately, learning from reward versus copying demonstrations.
2 · What a model is for
Practise in your head, or think before you act
Background planning
Between real steps, the agent replays imagined experience from its model and updates its policy or values as if it had really happened. Dyna (1990) did this with a lookup table; Dreamer does it with imagined rollouts in a learned latent space. Acting stays fast: at decision time the policy just answers.
Decision-time planning
When a decision is due, the agent uses its model to look ahead and picks the best-looking action: MPC for the drone in part 1, tree search in part 2, sampled action sequences in TD-MPC2. Decisions get better and slower. MuZero does both: it searches at every move, and trains its network on the results.
The terms come from the standard RL textbook. Both are “planning”; they differ only in when the model is consulted.
3 · What the model buys you
Fifty imagined steps for every real one
The classic demonstration is Dyna-Q in a small maze. The agent learns with Q-learning (part 5's update). After every real step it also records what happened in a model, a table of “from here, this move led there”, and replays n remembered transitions from that table, updating Q as if they were new. With n = 0 it is plain model-free Q-learning.
Lab 2 · Dyna maze
reward 1 only at G; γ = 0.95, α = 0.1, explore 10% of the timeSteps per episode, average of 10 runs
The curves are computed in your browser when the page loads: 10 runs of 30 episodes for each n. The shortest path is 14 steps; with 10% random moves the agent settles around 16–17.
Real experience is the expensive thing in robotics, and the model squeezes more learning out of each real step. Published results show the same direction at larger scale:
- PILCO (2011) learned to swing up and balance a real cart-pole with about 17.5 seconds of total experience, using a probabilistic model of the dynamics.
- SimPLe (2019) trained Atari agents inside a learned video model and set up the “Atari 100k” test: 100,000 agent steps, about two hours of play per game.
- EfficientZero (2021), built on MuZero, reached 194% of human mean score on Atari 100k. The authors note this is close to DQN at 200 million frames, with about 500 times less data.
4 · What the model costs
Small errors grow with every imagined step
The maze is easy because its model is exact: a table of what really happened. A learned model is a little wrong everywhere, and each imagined step starts from the previous imagined step, so errors compound. Worse, a policy trained in the model will hunt for the model's mistakes, because a mistake that predicts high reward looks like a great strategy. This is model exploitation. The 2018 World Models agent learned to exploit flaws in its own dream until the authors made the dream noisier.
Lab 3 · Roll the model forward
a swinging weight; the model's swing rate is off by a small percentageModel-based methods spend much of their design effort on this. MBPO (2019) branches only short imagined rollouts off real states, so errors have little room to grow. PILCO and many successors keep track of the model's uncertainty and avoid trusting it where it hasn't seen data. MuZero sidesteps pixel errors by never predicting pixels: its model is trained only to predict reward, value and the search policy. And every decision-time planner re-plans from the real state at each step, as the drone did.
5 · Side by side
The trade-offs
| Model-free RL | Model-based RL | Imitation | |
|---|---|---|---|
| real data needed | a lot | much less | demonstrations, but no trial and error |
| compute | low per decision | model training, plus planning if done at decision time | low |
| main failure | slow, unstable training | model errors and exploitation | small mistakes lead to unfamiliar states and grow |
| needs reward? | yes | yes (planning-only control needs a goal or cost instead) | no |
| reuses knowledge | tied to the task's reward | the model can serve new goals | tied to what was demonstrated |
Common slip: “behaviour cloning is model-free RL”
Behaviour cloning has no model, so it's model-free, but it isn't reinforcement learning: it copies actions and never sees a reward. Its known weakness follows from that. A small mistake takes the robot somewhere the demonstrator never went, where it has no example to copy, and errors snowball. DAgger (2011) fixes this by letting the robot drive and asking the expert to label the states it actually reaches. Many VLAs are trained by imitation and then improved with RL.
6 · Where language models fit
RL on a language model is model-free, because the “world” is known
When a language model is tuned with reinforcement learning (RLHF with PPO, or GRPO for reasoning), each action is the next token and the state is the text so far. The transition is trivial and exact: st+1 = st + token. Like the rules of Go, there's nothing to learn about it. The hard part is the reward: a learned reward model or an answer checker that scores the finished text.
So these methods are model-free in the RL sense. They're also not world models in this series' sense. The world model is needed only when the agent acts on something whose response it can't compute: a physical scene, other drivers, a user.
Prompted by the model-free versus model-based discussion in the YC Decoded video “World Models: An intuitive introduction” (July 2026), minutes 13 and 42–45. The rest is drawn from the sources below.
Sources. Reinforcement Learning: An Introduction, ch. 8 · Dyna (1991) · Model-based RL: a survey (2023) · PILCO and Gaussian-process control (2015 review) · SimPLe (2019) · EfficientZero (2021) · MBPO (2019) · MuZero (2019) · TD-MPC2 (2023) · DreamerV3 (2023) · DAgger (2011) · RLHF for instruction following (2022) · GRPO (2024)