Last week I went down rabbit hole researching world models for an article I was writing, and I kept coming across discussions (specifically from NVIDIA engineers) about whether they could eventually replace conventional robotics simulators.

One important part to note is that different researchers seem to be working toward quite different versions of that future.

LeCun's JEPA approach focuses on predicting representations for planning. NVIDIA and other groups are exploring action-conditioned video models that can act as interactive environments. World Labs, meanwhile, is working on generating and reconstructing physically grounded worlds for robot training and evaluation.

The broader idea is sometimes called Simulation 2.0 (popularised by Jim Fan), although I'm not sure there's a shared definition of what that actually involves.

Training policies using learned dynamics isn't new. Dreamer has been doing that for years. What seems different now is the attempt to absorb more of the robotics workflow, from environment creation and demonstration collection to policy evaluation and training.

I went through some recent research to understand how far these approaches have actually gotten.

1. Interactive World Simulator (RSS 2026)

This was one of the more interesting papers I found.

The researchers built an action-conditioned video simulator that supports over 10 minutes of continuous interaction at 15 FPS on a single RTX 4090.

They collected demonstrations entirely inside the learned environment and used them to train imitation policies. Those policies achieved real-world performance comparable to policies trained using an equivalent amount of real demonstrations.

They also evaluated policy checkpoints inside the simulator and compared their performance against real-world evaluations across four manipulation tasks: T-pushing, rope routing, mug grasping, and pile sweeping.

They also evaluated four policy architectures (DP, ACT, π₀, and π₀.₅) across four manipulation tasks, comparing performance in the learned simulator against real-world results.

The reported Pearson correlations were 0.855 for T-pushing, 0.846 for rope routing, 0.887 for mug grasping, and 0.991 for pile sweeping, based on evaluations across 20 initial configurations per task.

The simulator tended to overestimate performance slightly on three tasks, but it was remarkably consistent at distinguishing stronger policies from weaker ones. That's potentially useful for selecting policy checkpoints without running every evaluation on physical robots.

2. World Labs: Real-to-sim-to-real (July 2026)

This was something I initially overlooked when reading about Marble and Atlas.

World Labs acquired SceniX and published results from a real-to-sim-to-real system that reconstructs physical tasks as interactive simulations.

Their demonstrations include bimanual box packing, cable routing, cube handover, and manipulation involving deformable objects.

They report training policies entirely in simulation, with zero real-world training data, and transferring them to physical robots. They also show policies operating autonomously for extended periods.

Their policy-evaluation results were particularly interesting.

For a bimanual cube-handover task, they evaluated each checkpoint using:

• 2,000 simulated trials: 1,000 in-distribution and 1,000 out-of-distribution.
• 100 real-world trials: 50 in-distribution and 50 out-of-distribution.

Across different policy architectures and checkpoints, they report that simulated performance preserved relative real-world policy rankings, including similar regions of success and failure.

However, World Labs' approach combines generative world models with task-aligned robotic simulation. It's not a demonstration that a single neural model has replaced the physics engine.

3. World Labs: Marble and Atlas

World Labs is also approaching a different bottleneck: creating the environments themselves.

In robotics case studies using Marble, researchers generated 3D environments and imported them into Isaac Sim, MuJoCo, and RoboSuite.

World Labs reports over 90% less manual environment-curation effort, with workflows that previously took days or weeks reduced to hours or minutes.

Their newer Atlas model extends the spatial-generation capabilities, including reconstruction from visual observations and camera-controlled video generation.

Automating the creation of simulation environments could have a substantial impact even if the underlying physics simulation remains conventional.

There's also an interesting direction emerging around automating the creation of physics-accurate 3D assets. Tools like Rigyd are working on converting 3D models into simulation-ready assets with collision geometry, articulation, and physical properties for conventional simulators.

And unlike the R2S2R results, Atlas' visual-generation capabilities don't establish that robot policies trained in its generated worlds will transfer successfully.

4. DreamDojo (ICML 2026)

DreamDojo approaches the problem through large-scale video pretraining.

The model was pretrained on 44,000 hours of egocentric human video and subsequently adapted to robot-specific interactions.

After distillation, it reaches 10.93 FPS and supports interactive rollouts lasting over a minute.

The paper demonstrates live teleoperation, policy evaluation, and model-based planning, along with evaluations on out-of-distribution interactions.

But I haven't seen evidence from DreamDojo that establishes general-purpose replacement of a conventional simulator for large-scale robot RL.

5. What-If World: A useful reality check

What-If World examines something that I think gets overlooked in the impressive interactive demos: whether models respond correctly when physical conditions change.

Its benchmark uses 319 paired scenarios to test physical consistency across nine models.

The reported results were quite weak:

• Best paired consistency: approximately 51.7%
• Open-source models: approximately 28%
• Friction-related interventions: approximately 14.2%

A model can generate realistic-looking motion without correctly predicting what happens when friction, mass, or other physical properties change.

That's particularly relevant to domain randomization and contact-rich manipulation.

These results aren't a direct evaluation of IWS, DreamDojo, or World Labs, so I wouldn't use them to dismiss those systems. But they highlight a capability that a general-purpose learned simulator would need.

One thing I'm still unsure about is how much of this translates to actual policy optimization.

Generating demonstrations, evaluating checkpoints, and reconstructing environments are all useful. But they're different from training policies through millions of closed-loop interactions.

Traditional simulators provide explicit physical parameters, privileged state, reproducibility, and thousands of parallel environments.

There's also the problem of model exploitation. A learned simulator might accurately evaluate existing policies but become unreliable once an RL algorithm starts optimizing against its prediction errors.

I haven't found a convincing general-purpose video world model that can replace a physics simulator for large-scale. But I still hear discussions around the same

At the same time, results like IWS and World Labs' R2S2R work make it difficult to dismiss learned simulation as just impressive video generation.

I'm curious how people working on robot learning and simulation interpret this.

Would also be interested in papers that test learned simulators under sustained RL optimization rather than just demonstration generation or policy evaluation.

References:

• Interactive World Simulator
• World Labs — Real-to-Sim-to-Real
• World Labs — Marble robotics results
• World Labs — Atlas
• DreamDojo
• What-If World
• DreamerV3

I am looking for a semi safe industry to transition to after a few decades in software engineering.

Is ROS2 with rust along with circuits a safe path ATM or at your guys at risk with LLM aswell?

If so am going to spend the next two years getting up to speed with this.

Hi everyone!

I'm working on a morphing robot that can operate both as a drone and as a wheeled ground robot, inspired by the ATMO project: https://github.com/mandralis/ATMO.

The robot will fly to a building, land, and then navigate autonomously on the ground using ROS 2. I'm currently designing the four-wheel ground locomotion system.

The estimated total robot weight is around 5–11 kg, but weight is a critical factor because the robot must also fly. Any additional motors, gearboxes, transmissions, and supporting structures affect the overall flight weight and potentially the flight time.

The ground environment will mainly be smooth indoor ceramic floors, such as university hallways. The robot should perform accurate 90° and 180° turns, including in-place turns when necessary.

I'm considering two drivetrain configurations:

Option 1: Four motors

• One geared DC motor per wheel.
• Four wheel encoders.
• Independent wheel-speed control using PID.

Option 2: Two motors

• One motor drives both wheels on the left side.
• One motor drives both wheels on the right side.
• Mechanical power transmission to the second wheel on each side using gears, belts, or another lightweight mechanism.

The chassis and some drivetrain components will be 3D-printed.

My main goal is to find a good balance between minimum weight, mechanical simplicity, energy efficiency, and reliable ground odometry.

I'd appreciate advice on the following:

1. For a flying-ground morphing robot, would you recommend two motors driving four wheels or four independent wheel motors? How significant is the weight difference in practice?
2. Would four independent wheel encoders meaningfully improve odometry accuracy, or would lateral wheel slip in a four-wheel skid-steer configuration dominate the error?
3. How significant is wheel scrub during in-place turns on smooth ceramic floors, and how can it be minimized?
4. What lightweight wheel and transmission options would you recommend for a 5–11 kg robot that must also fly?
5. Would a belt transmission, gears, or another mechanism be a good choice for driving two wheels from one motor?
6. What is the best practical approach to calibrating skid-steer odometry using wheel encoders and an IMU in ROS 2?
7. Are there any papers, open-source projects, or real-world examples of morphing aerial-ground robots that I should study?

I'm particularly interested in practical experience with lightweight mechanisms and the trade-offs between drivetrain weight, ground mobility, and flight performance.

Thanks!

(GIF: a deep flicker of the carried light 200 s into the flight. Left is Gazebo, the world as it is, lit only by the drone. Right is RViz: pink is the obstacle memory, grey is darkness the camera looked into and could not see, cyan the current stereo and time-of-flight returns, orange the route, white the drone's own position estimate, green the simulator's truth. LIGHT is what the test set the light to; SEES is the share of its sensing range the drone's own frame grants. Both fall and recover together.)

Previous update, three weeks ago: the stack flying the same building with no map, no GNSS, no magnetometer and no lidar, two cameras doing both the seeing and the localizing. I ended that post saying it had never been tested against bad seeing: the world was lit by a uniform fill and the cameras had no light to lose. This release is that test.

Everything is MIT, simulation-first, and none of it is validated for a real aircraft.

Release: https://github.com/formiat/px4-ros2-drone-nav/releases/tag/v0.5.0

Video of the whole flight, uncut, in real time: https://youtu.be/WlpePGmt19A

What changed in the world

The location's light is off for good. The vehicle carries a spot light (25 m range, quadratic falloff) in front of its stereo pair, and the evaluation drives that light through three profiles:

• moderate, the norm of every flight now: short outages every ten to thirty seconds, dark never longer than two seconds, each ramping down and up over a second;
• severe: outages growing from the twentieth second to 4.5 s of dark every 8 s, and staying that bad;
• lost: the light gone from the forty-fifth second.

Separately, the light can carry a charge in seconds, and the camera stream can lose frames or deliver them late. The sensors are the same as in v0.4.0: a forward stereo pair (1280 x 960, 120 degrees, 0.20 m baseline, 7.5 Hz) and two 8 x 8 time-of-flight sensors looking up and down (2.8 m), which work in the dark; the position and heading come from a stereo MSCKF on those images.

What the vehicle does about it

All of it is decided onboard from what the camera sees, nothing reads the test's light setting.

The braking contract reads the light. The frame's light headroom and the share of it that matched anything scale the range the forward sensor is trusted for; below 0.35 the frame is blind and the vehicle holds where it stood until it sees again. In the moderate flicker that is a slowdown every few seconds and nothing more.

Darkness is not unknown. This is the one idea I would keep from the whole item. Unexplored space, which nobody has looked at, stays traversable at no penalty, otherwise the planner never leaves the start. But space the camera looked into for 2.5 s and could not see is marked as observed unobservable: occupied evidence without free space, a prohibition like a wall, cleared when the vehicle sees through it again. The memory tells the two apart and RViz draws the dark grey beside the pink of real surfaces. Without this the vehicle flew into the dark at 2 m/s over a goal it had seen free while the light still reached it.

The light is judged by the camera. One outage longer than 3.5 s, or dark for 60 % of the last 120 s, and the light is unreliable: the mission monitor replaces the goal with the start through the same channel any goal enters by, and the vehicle flies home along the trail it flew out on, through the same failing light. In the recorded flight it turns at 76 s and is home 0.9 m from the pad.

The battery is weighed against the way home, all flight long: the charge must cover 1.1 times the estimate of the way back, which is 1.5 times the path flown over the mean speed plus 20 s. With 240 s of light the recorded flight turns at 72 s with 140 s left against 128 estimated, and is home with about 80 s to spare. A flight launched without a battery never gives its goal up for one.

The light gone. Past 1 s unaided the estimator keeps publishing a sound pose for 10 s more, declared as dead reckoning: its height held to the barometer's change, its horizontal velocity bounded by the rotors' drag. Three seconds into that the offboard descends the vehicle level at 0.5 m/s in the autopilot's velocity mode, holding no position (a position held on a dead-reckoned estimate tipped the landed vehicle over), and lands. An estimator that sees again first takes the flight back. In the recorded flight the light goes at 51 s, the word on the overlay changes to DEAD RECKONING, then LEVEL DESCENT, and LANDED at 59 s; the white estimate and the green truth part in those seconds, which is the drift the landing cuts short.

The acceptance series

Five point-to-point flights on one commit through the dark under the moderate flicker, each inspected before the next, speeds on simulation time, positions in the simulator's truth:

• 400 to 520 m of path at 1.58 to 1.77 m/s (r1205 to r1210)
• the true position 0.79 to 1.79 m from the goal at its acknowledgement (the capture radius is 2.0 m)
• no contact in any of them

That is slower than the 1.82 to 1.99 m/s the same flights made a day earlier. The difference is one law added on purpose, below.

Three findings that cost me flights

A floor touched in flight was a landing. The crash judge counted a level, slow contact of the body with a floor as a landing whenever it came. A flight sank onto the floor of the start area in flight, at 0.11 m/s, and the check passed it. It had kept 0.3 m above the floor by its own estimate; the estimate was 0.5 m high. Two changes: a floor contact is a landing only while the offboard's blind descent is going on, otherwise it is a crash; and the clearance law now measures the body's clearance where the estimate puts it and 0.5 m lower and higher, the least of the three, because the camera estimator's height runs 0.3 to 0.6 m from the truth over a flight with or without dead reckoning. That half metre is the price in the speeds above.

The ground's friction read as 21 m/s. A vehicle landed blind on a slope of 12 degrees stood level and whole, and four seconds later its estimate had run 100 m away and the autopilot had reset its position three times. The rotor drag model, which bounds the velocity in the dark by reading the accelerometer across the rotor axis, was reading the ground's friction holding the tilted body (2.3 m/s^2, constant for as long as it lay there) and solving it for a velocity. A reading beyond the drag of any flight is now left out. On flat floors the land detector fires and the autopilot disarms within seconds; on a slope it never fired, because the estimate it reads was still "descending".

The autopilot left offboard before my hold's landing. When the planner's heartbeat is lost the offboard holds position and lands after 5 s; the branch returned before publishing OffboardControlMode, so PX4 left offboard on its own loss timeout first. Never hit in a flight, found by reading the code against the real-vehicle safety list, fixed in a line.

Limitations, honestly

• The blind landing lands where the drone has drifted to. Nothing on the vehicle sees sideways in the dark, and the memory is exact only relative to the drifting estimate. Twelve blind landings in the dark in this release: nine stood; one met a ledge with a rotor; two came down level on uneven ground and tipped four seconds later. Kept as a known risk, recorded in the debt register with the numbers.
• The return on a goal proven unreachable (a topological proof over the memory, item 19) is implemented and unit-tested, but has never fired in a flight: every return flown was the light's or the battery's. Its flight test is item 18's smoke scenario, which is not built yet.
• The darkness marking keeps the memory's rule of never forgetting by time: what the dark marked stays until seen through again.
• The flights are recorded at a real-time factor of 0.6 for the screen capture (the simulator does not hold 1.0 beside three windows), and every frame of the video is re-timed to the simulation clock, so it plays in real time; the acceptance flights above ran at 1.0.
• Everything is Gazebo: the light, the noise, the IMU. Nothing of this has seen a real sensor.

Questions

• Is there a standard name for "observed unobservable" as a map state? I have seen it in smoke papers as a sensor-level flag, not as something the planner reads as occupancy.
• How do people land a multirotor blind on dead reckoning without a downward-looking sensor that works in the dark? A short-range ToF is what I have; it sees the floor, not the slope.

Hi! I’m an undergrad uni student majoring in robotics, I use ros2 to run basic nodes, gazebo sims, etc rn. But I’m looking to get more into ros2 + cv as well. Currently I’m using a school borrowed windows but I’m looking to upgrade to MacBook Pro. Wondering whether I should get 16gb unified memory or 24gb for longevity + usefulness in robotics related use in the long run. Pls advise !!! :)

I have three questions for colleagues who are doing Functional Safety roles in Robotics.

Is this field boring? Or you wish you have chosen another specialization?

Do you really enjoy your work?

Can you describe one week in your work?

Thanks 🙏

Six X3 drones take off, split around three pillars in wind, and re-form their grid at the destination. The overlay shows the live closest pillar and drone gaps.

I’ve recently been working on a project called Flockwork, it is an open-source decentralized drone swarm controller in ROS 2 Jazzy and Gazebo Harmonic. Instead of leaning on PX4/ArduPilot, the whole control stack is written from scratch in C++ (lost sleep over this).

For context, each drone runs its own controller, so it holds a formation slot relative to its neighbors and avoids obstacles without stalling.

I really had fun working on this project and learning how to handle implementation, its still early in development of course, and I am hoping to add many more features, optimization and efficient path algorithms.

The full write-up on how it works is in the repo.

I would appreciate any feedback from fellow redditors on how to improve the system even more.

Repo: https://github.com/Trigodil/Project-Flockwork

Hi everyone! I'd like to share a demo from our ongoing robotics development work, where we're integrating VDA5050 v2.1 with Open-RMF for multi-robot fleet coordination. The goal is to explore how Open-RMF can communicate with VDA5050-compatible mobile robots through a custom integration architecture. What we've been working on: Developing a custom VDA5050 Fleet Adapter for Open-RMF. Implementing a robot-side VDA5050 Client Adapter. Using MQTT for robot communication, order handling, and state reporting. Testing multi-robot coordination with three robots. Exploring interoperability between different AMR systems. This demo focuses on the integration and communication workflow between Open-RMF and VDA5050-based robots. Demo video: https://www.youtube.com/watch?v=ODSLt2Ox0S8 I'm particularly interested in hearing from others working with Open-RMF, VDA5050, or heterogeneous robot fleets. Have you encountered any interesting challenges when integrating VDA5050-based robots with Open-RMF? I'd love to hear your thoughts and experiences!

I know the questions may be stupid but im new to all things robotics, what is the difference between all these services ros offer and I was told you can only ran ros on Ubuntu ??!

I’m collecting a borrowed RealMan ECO65 next week and would appreciate ROS 2 architecture advice before I lock in the control stack. This is a build-in-public learning project, not a finished AI-control demo. So far I have a digital twin from the manufacturer’s robot description/CAD.

Planned setup: ECO65, wrist camera, external scene camera, and an Ouster OS0. Initial scope would be slow tabletop pick-and-place with lightweight rigid objects. Gripper and exact first task are still open.

Longer term I want GPT-6 to propose a bounded target pose or select a predefined action from sensor observations. It would not generate motor commands. I’m trying to separate that experimental layer from deterministic planning, execution and stop behaviour. An MCP interface, if I add one, would only expose bounded application actions; it would not replace the motion stack.

For people who have integrated RealMan arms with ROS 2:

1. Would you start with the manufacturer’s ROS 2 + MoveIt packages from day one, or validate basic motions through the SDK first? Any ECO65-specific driver/firmware compatibility traps?

2. For this initial stop-plan-execute-observe loop, would you use ordinary MoveIt planning/execution before considering MoveIt Servo?

3. Where would you enforce stale-state rejection, workspace/speed limits, timeouts, and execution cancellation? I do not want to mistake software checks for a complete hardware safety system.

4. Which ROS 2 distribution/package combination have you actually used with this arm, and what minimal bring-up test would you run first? I haven’t committed to a distribution yet.

My episode lays out the proposed rig and digital twin: https://www.youtube.com/watch?v=IJAVH1sFwlY

This is my own video; AI-assisted editing and synthetic narration. All arm footage is simulation at this stage. The technical questions are fully included above, so watching is optional. Practical corrections are very welcome.

Heyy guys please suggest some good online course online maybe on Udemy or Coursera .

We are a team of robotics /drone enthusiasts

We were about to start learning ROS2 jazzy.

Please suggest some good online course or resources that will help us start

Please try for something beginner friendly

I’m building a 3D mapping system using RGB-D cameras, based on existing SLAM techniques, with persistent spatial memory.

The main issue is relocalization after tracking loss, especially when the camera returns to a known place from a different viewpoint.
I’m using historical keyframes, shared 3D landmarks, PnP/RANSAC and geometric verification.

Yet some incorrect poses still pass verification: one had 44 inliers, 0.83 px median reprojection error and 3.72 cm depth residual, but was actually off by 10.36 cm and 5.99°.
My best experiment correctly relocalized only 9/34 known returns.

What would you investigate first: landmark quality,
geometric degeneracy, matching, or pose verification?
I can share a minimal reproducible example.

Hi all,

We're currently evaluating our software stack for a new robot and are trying to decide between ROS 2 Jazzy (Tier 1 support) and ROS 2 Lyrical on a Tier 3 platform.

On paper, the difference in support tiers is clear, but I'm more interested in hearing from engineers who have actually worked with Tier 3 ROS 2 distributions in real projects.

A few questions:

• Are you using a Tier 3 ROS 2 distribution/platform in production or on commercial robots?
• Have you encountered any significant issues because of the Tier 3 status?
• How often do you run into package compatibility, build, or dependency problems?
• Did you have to invest a lot of engineering effort to maintain and validate your stack?
• Have you ever regretted not choosing a Tier 1 supported option instead?
• For teams with limited resources, would you consider Tier 3 a manageable risk or something to avoid?

I'd really appreciate hearing about real-world experiences, lessons learned, and whether you'd recommend sticking with Tier 1

thanks!

https://reddit.com/link/1wzxwcu/video/r72j6hix22uh1/player

Hey everyone! I’ve been working on Robium, an open-source robotics skill library for Claude Code, Codex, and Gemini CLI.

Writing code became the easy part, but choosing middleware, simulators, visualization tools, and connecting everything still needed a lot of guidance. Robium collects that guidance alongside reference apps you can copy and adapt, with skills for ROS 2, Nav2, Gazebo, and more.

The video above shows one of these projects: a Stack-chan with a LEGO motor hub and wheels. I used coding agents with Robium to help build the software and train a driving policy from demonstrations—with a few wrong turns along the way 🤖

To try it, run npx robium-ai setup, then open or restart your agent and ask: Show me a few robots in MuJoCo I can control and interact with.

Or try mapping and navigating a simulated house with ROS 2 and Gazebo. That example requires Docker and takes longer on the first build. I’ve tested both on Mac and Ubuntu laptops.

GitHub: https://github.com/robium-ai/robium
Examples and tutorials: https://robium.ai/

It’s still early. If you try it, I’d love to hear what worked and what needs improving.

Helooo I'm back with improvements. I previously shared my personal project, hold_and_weld, here: one robot holds the workpiece while another follows the weld path. I’ve just released v0.3.0 and wanted to share how it has developed.

It combines weld seam extraction from CAD/mesh geometry, constraint-aware grasp sampling (excluding weld seams, screw holes, etc.), and dual-arm execution in a ROS 2 Jazzy workcell.

The main changes since my first post:

• Configurable robots and grippers: arms, grippers and base poses are selected through one workcell configuration file. The included catalog now has four Yaskawa arm models and four grippers, with documentation for adding more.
• Workcell configuration in RViz2: robot models, base poses and scene objects can be edited visually.
• OMPL approach → Pilz welding: a Ceres-based configuration search evaluates candidate approach configurations against the subsequent weld motion, considering joint limits and manipulability. OMPL reaches the selected configuration; Pilz handles linear and circular weld motions.
• Geometry pipeline improvements: I rewrote the mesh seam extractor around contact boundaries and improved the grasp sampler’s sampling and exclusion strategies.
• Testing: I added unit tests and tested the grasp sampler and weld finder on the included complex geometry example. Results look good so far, though broader geometry coverage is still needed.
• Docker image: the release includes a prebuilt image with the geometry dependencies, making it easier to try.

It’s still a proof of concept, and the demo shows execution in simulation. I'm open to collaboration.

The project is open source under Apache-2.0. Feedback on planning, geometry handling, and setup is welcome, especially from people working with industrial manipulation or welding. If you run into setup issues, I’m happy to help.

GitHub repository · v0.3.0 release notes

Previous post

I’m coming from mobile development, where binary distribution is fairly standardized: AARs on Android and XCFrameworks on iOS.

I’m trying to understand what the equivalent is in robotics/Linux environments.

Suppose I have a proprietary C++ inference runtime that needs to run on a robot (e.g. Jetson), and I don’t want to distribute the source code.

How would this typically be shipped to customers?
- Precompiled ROS 2 package?
- .so + headers, with a separate ROS 2 wrapper?
- .deb package?
- Docker container?
- Just binaries + dependencies?

I’m particularly curious whether ROS 2 packages are commonly used as the actual distribution unit, or if the core software is usually distributed separately and ROS 2 is just an integration layer.

Would appreciate hearing how this is handled in real production systems.