What world models still get wrong

World models · 2026 · Part 6 of a series that starts with The drone and the world model

The open problems, one at a time, each with a lab or a worked number, and with opinion labelled as opinion.

World models · part 6

What World Models Still Get Wrong

Ask a video world model what happens if a car steers into a house, and it may show the house turning into a road. Why do models that look so real fail at exactly the moments that matter?

This page goes through the open problems one at a time: rare events, precision, memory, adapting on the spot, and the senses robots still lack. Each has a lab or a worked number. Where a shortcut version is commonly said, it's marked COMMON SLIP. Several sections are researchers' opinions rather than settled results, and they're labelled as such.

1 · Rare events

The house that turns into a highway

Nearly all driving video shows cars staying on the road. A model trained on it learns, above all, that roads continue. Condition it on “steer hard right, into the house”, an action it has almost never seen, and the most familiar continuation wins: the house gets redrawn as more road. The crash, the one outcome a planner most needs to predict, is the one the model knows least.

One fix is mini-batch composition: choosing what goes into each training batch on purpose, so rare cases appear often enough to learn from.

Lab 1 · Build a batch

grey = an ordinary road clip, red = a crash clip

crash clips in the data: 1 in

10,000

batch size =

256

oversample crashes ×

1

Draw another batch

crash clips per batch

0.03

batches between crashes

39

share the model sees

0.01%

Common slip: calling this the “PINN” problem

PINNs, physics-informed neural networks, are a specific technique: they add the error of a known physics equation to the training loss, so the network is punished for breaking, say, conservation of momentum. That is one proposed remedy for models that ignore physics. The failure itself is a data-distribution problem, and batch composition, reweighting and simulated crash data are the other remedies.

2 · Precision

A network can't even fit a sine wave exactly

64-bit floating point carries about 16 significant digits, so a physics simulator's arithmetic is good to roughly 10⁻¹⁶. A neural network trained by gradient descent on something as simple as a sine wave gets close quickly and then stalls far above that. Small errors in a world model add up over a long imagined rollout.

Lab 2 · Fit sin(x)

a 1–16–16–1 tanh network, Adam, 64 points, 64-bit numbers, live in your browser

Train Reset

This is a practical limit, not a law. With 64-bit numbers and a second-order optimiser, small networks can be pushed much lower, and nobody needs 16 digits for most tasks. The question researchers argue about is how much precision long imagined rollouts need before their errors compound into nonsense.

3 · Memory

A transformer keeps every moment instead of a summary

Standard attention compresses nothing over time. Every past token's keys and values stay in the KV cache, so memory grows in a straight line with the length of the context. For a model with 32 layers and 8 key-value heads of 128 numbers each, stored in 16 bits, that is 2 × 32 × 8 × 128 × 2 bytes = 128 KiB per token: about 128 GiB for a million tokens. A world model that runs for hours needs a summary that stays the same size. Linear attention and state-space models are one route: they fold the past into a fixed-size state.

4 · Adapting on the spot

The floor changed. Notice now, not after retraining.

A tennis player adjusts to a new court within a few points. A robot meets a new surface, a damp tabletop, a heavier cup, all day long. Its world model's numbers, such as friction, have to update from the last few seconds of experience, during the task. A model that only changes during a nightly retraining run keeps predicting the old world.

Lab 3 · Slide a puck

each push: launch speed known, stopping distance measured with noise; friction jumps at push 20

Run Reset

Each push gives a noisy reading of friction, μ = v² / (2·g·d). The slow learner moves 3% of the way towards each new reading, like a large model fine-tuned with a small learning rate. The fast estimator averages the readings with recent ones weighted most (weights fall by 0.6 per push), a standard trick in online system identification. Fast adaptation has a price: it is noisier when nothing has changed.

Planning also has to fit in real time. Part 2 showed tree search spending a fraction of a second per move with hundreds of network calls. People seem to run a fast habitual controller and a slower deliberate one side by side, and the slow one can interrupt the fast one mid-movement. How to build that split well in a robot is open.

5 · Opinion, labelled as such

The squint test

A common objection to copying the brain: aeroplanes don't flap their wings. The reply is a squint test: squint at a bird and a plane, and both still have two wings and a tail. The details differ but the structure is shared. By that test, a fast policy plus a predictive world model plus planning at decision time looks more brain-like than a language model predicting tokens one after another.

Some researchers go further and say current AI still fails the test in one big way: the brain keeps learning while it runs, and goes offline to consolidate. Nearly every animal studied sleeps, despite the danger, and during sleep the hippocampus replays recent experience. Today's models are trained once, then frozen. This is a view about where the field should go, not an established result.

6 · The missing sense

Robots are nearly numb

Your skin reports pressure, sideways shear (the first sign that something is slipping), vibration, temperature and pain, all over your body. The hairless skin of one hand alone holds about 17,000 touch receptors. When fingertips are numbed in experiments, people grip too hard or too loosely and fumble small objects, even while watching their hands. Try tying a shoelace with numb fingers.

Most robot hands feel far less: a motor-current reading, perhaps a force sensor at the wrist or a few pads on the fingertips. Camera-based touch sensors and tactile skins exist, but they are not yet cheap, tough and widespread. A world model can only predict what its sensors measure, so this hardware gap limits manipulation however good the model gets.

The series so far

From a drone to the open problems

Plan with a model you trust (the drone). Search when the rules are known (search). Face moves you can't list and labels you can't afford (too many moves). Learn the world from video (watch, then practise). Predict the gist (latents). And know where it all still breaks.

Based on minutes 63–74 of the YC Decoded video “World Models: An intuitive introduction” (July 2026). The claims were checked against published work and labelled where they are opinion.

Sources. Physics-informed neural networks · Machine epsilon · Adam (2014) · Recursive least squares (online estimation) · Sleep across animals · Tactile units in the human hand (1979) · Mechanoreceptors

Next: Three programs