NVIDIA Cosmos

World models · 2025

The idea

Genie is a closed, playable world. What if you need open weights you can fine-tune on your robot or car?

NVIDIA Cosmos (Agarwal et al., Jan 2025, arXiv:2501.03575) is a platform, not one checkpoint. The paper’s claim is narrow and useful: Physical AI is starved of (observation, action) data because exploratory actions in the real world are dangerous. A world foundation model is a generalist video predictor you post-train into a specialist - a digital stand-in you can safely poke.

The recipe in the paper:

1. Curate video. From a 20 million hour collection, keep about 100 million clips of 2–60 seconds with rich dynamics; caption with a VLM.
2. Pretrain two families. Transformer diffusion (continuous tokens) and transformer autoregressive (discrete tokens), plus a learned video tokenizer - because naive video token counts make training impossible.
3. Post-train small. Camera-pose control, robot video-action prediction, driving. The generalist does the heavy lifting; the target dataset can stay small.

NVIDIA later split the product tree into Predict (future video), Transfer (world-to-world, depth/seg/maps as control), and Reason (language-side physical common sense). Later Predict2.5 / Transfer2.5 releases keep the same job: builders post-train. Licenses on the original Predict1 line are permissive (NVIDIA Open Model License).

What the paper says a WFM is for: policy evaluation, policy initialization, imagination-style policy training, planning / MPC, and synthetic data (including Sim2Real when conditioned on depth or semantics). Notice the honest ranking: most of those uses are cheaper than the real world, not equal to it.

The misconception to name

It is tempting to read “world foundation model” as “we solved physics at foundation-model scale.” Section 8 of the paper is the opposite:

Current models, including ours, fall short as reliable simulators of the physical world.

They list object impermanence, bad contact dynamics, instruction inconsistency, and generated video that looks real while violating gravity, light, or fluids. That is the same residual Genie 3 carries, with one difference: Cosmos ships weights so you can measure the failure on your domain.

Verdict

What it fixedAn open world-foundation-model platform for Physical AI: pretrain on curated video, then post-train for driving, robots, or camera control. Weights and tokenizers under a permissive license.

What it still costsNVIDIA's own paper: still not a reliable physics simulator. Object permanence, contact-rich dynamics, and gravity can fail. The product is a data engine and policy gym, not a digital twin of reality.

The story, animated

Curate, pretrain three artifacts, fan out to three targets - then the paper’s own honest admission.

Where it sits

Worth having first: Three programs · Genie 3

This is the generative-simulator column. Contrast V-JEPA 2-AC / LeWorldModel (latent planners) and Dreamer 4 (imagination training with a generative latent model you do not ship as a public gym).

Next wave on this track: language and digital world models - the third column.

Source-checked against Agarwal et al., “Cosmos World Foundation Model Platform for Physical AI” (arXiv:2501.03575), abstract, §1–2 and limitations. Product split Predict / Transfer / Reason from NVIDIA Cosmos docs.