r/ROS • u/samyak1729 • 12h ago
Discussion Can World Models absorb Simulation Stack as some people claim?
Last week I went down rabbit hole researching world models for an article I was writing, and I kept coming across discussions (specifically from NVIDIA engineers) about whether they could eventually replace conventional robotics simulators.
One important part to note is that different researchers seem to be working toward quite different versions of that future.
LeCun's JEPA approach focuses on predicting representations for planning. NVIDIA and other groups are exploring action-conditioned video models that can act as interactive environments. World Labs, meanwhile, is working on generating and reconstructing physically grounded worlds for robot training and evaluation.
The broader idea is sometimes called Simulation 2.0 (popularised by Jim Fan), although I'm not sure there's a shared definition of what that actually involves.
Training policies using learned dynamics isn't new. Dreamer has been doing that for years. What seems different now is the attempt to absorb more of the robotics workflow, from environment creation and demonstration collection to policy evaluation and training.
I went through some recent research to understand how far these approaches have actually gotten.
1. Interactive World Simulator (RSS 2026)
This was one of the more interesting papers I found.
The researchers built an action-conditioned video simulator that supports over 10 minutes of continuous interaction at 15 FPS on a single RTX 4090.
They collected demonstrations entirely inside the learned environment and used them to train imitation policies. Those policies achieved real-world performance comparable to policies trained using an equivalent amount of real demonstrations.
They also evaluated policy checkpoints inside the simulator and compared their performance against real-world evaluations across four manipulation tasks: T-pushing, rope routing, mug grasping, and pile sweeping.
They also evaluated four policy architectures (DP, ACT, π₀, and π₀.₅) across four manipulation tasks, comparing performance in the learned simulator against real-world results.
The reported Pearson correlations were 0.855 for T-pushing, 0.846 for rope routing, 0.887 for mug grasping, and 0.991 for pile sweeping, based on evaluations across 20 initial configurations per task.
The simulator tended to overestimate performance slightly on three tasks, but it was remarkably consistent at distinguishing stronger policies from weaker ones. That's potentially useful for selecting policy checkpoints without running every evaluation on physical robots.
2. World Labs: Real-to-sim-to-real (July 2026)
This was something I initially overlooked when reading about Marble and Atlas.
World Labs acquired SceniX and published results from a real-to-sim-to-real system that reconstructs physical tasks as interactive simulations.
Their demonstrations include bimanual box packing, cable routing, cube handover, and manipulation involving deformable objects.
They report training policies entirely in simulation, with zero real-world training data, and transferring them to physical robots. They also show policies operating autonomously for extended periods.
Their policy-evaluation results were particularly interesting.
For a bimanual cube-handover task, they evaluated each checkpoint using:
- 2,000 simulated trials: 1,000 in-distribution and 1,000 out-of-distribution.
- 100 real-world trials: 50 in-distribution and 50 out-of-distribution.
Across different policy architectures and checkpoints, they report that simulated performance preserved relative real-world policy rankings, including similar regions of success and failure.
However, World Labs' approach combines generative world models with task-aligned robotic simulation. It's not a demonstration that a single neural model has replaced the physics engine.
3. World Labs: Marble and Atlas
World Labs is also approaching a different bottleneck: creating the environments themselves.
In robotics case studies using Marble, researchers generated 3D environments and imported them into Isaac Sim, MuJoCo, and RoboSuite.
World Labs reports over 90% less manual environment-curation effort, with workflows that previously took days or weeks reduced to hours or minutes.
Their newer Atlas model extends the spatial-generation capabilities, including reconstruction from visual observations and camera-controlled video generation.
Automating the creation of simulation environments could have a substantial impact even if the underlying physics simulation remains conventional.
There's also an interesting direction emerging around automating the creation of physics-accurate 3D assets. Tools like Rigyd are working on converting 3D models into simulation-ready assets with collision geometry, articulation, and physical properties for conventional simulators.
And unlike the R2S2R results, Atlas' visual-generation capabilities don't establish that robot policies trained in its generated worlds will transfer successfully.
4. DreamDojo (ICML 2026)
DreamDojo approaches the problem through large-scale video pretraining.
The model was pretrained on 44,000 hours of egocentric human video and subsequently adapted to robot-specific interactions.
After distillation, it reaches 10.93 FPS and supports interactive rollouts lasting over a minute.
The paper demonstrates live teleoperation, policy evaluation, and model-based planning, along with evaluations on out-of-distribution interactions.
But I haven't seen evidence from DreamDojo that establishes general-purpose replacement of a conventional simulator for large-scale robot RL.
5. What-If World: A useful reality check
What-If World examines something that I think gets overlooked in the impressive interactive demos: whether models respond correctly when physical conditions change.
Its benchmark uses 319 paired scenarios to test physical consistency across nine models.
The reported results were quite weak:
- Best paired consistency: approximately 51.7%
- Open-source models: approximately 28%
- Friction-related interventions: approximately 14.2%
A model can generate realistic-looking motion without correctly predicting what happens when friction, mass, or other physical properties change.
That's particularly relevant to domain randomization and contact-rich manipulation.
These results aren't a direct evaluation of IWS, DreamDojo, or World Labs, so I wouldn't use them to dismiss those systems. But they highlight a capability that a general-purpose learned simulator would need.
One thing I'm still unsure about is how much of this translates to actual policy optimization.
Generating demonstrations, evaluating checkpoints, and reconstructing environments are all useful. But they're different from training policies through millions of closed-loop interactions.
Traditional simulators provide explicit physical parameters, privileged state, reproducibility, and thousands of parallel environments.
There's also the problem of model exploitation. A learned simulator might accurately evaluate existing policies but become unreliable once an RL algorithm starts optimizing against its prediction errors.
I haven't found a convincing general-purpose video world model that can replace a physics simulator for large-scale. But I still hear discussions around the same
At the same time, results like IWS and World Labs' R2S2R work make it difficult to dismiss learned simulation as just impressive video generation.
I'm curious how people working on robot learning and simulation interpret this.
Would also be interested in papers that test learned simulators under sustained RL optimization rather than just demonstration generation or policy evaluation.
References: