NVIDIA Launches Open-Source Robot World Model Trained on 44,000+ Hours of Human Video, Boosting Real-World Success Rates by 17%
Summary
NVIDIA launches DreamDojo, a fully open-source robot world model trained on over 44,000 hours of human video that boosts real-world robot success rates by 17%, enabling real-time control across thousands of tasks without traditional physics engines.
Key Points
- NVIDIA releases DreamDojo, a fully open-source robot world model that learns physics and real-world interactions from 44,711 hours of egocentric human video data, bypassing the need for traditional physics engines.
- DreamDojo uses a spatiotemporal Transformer VAE to extract continuous latent actions directly from pixels, allowing the model to translate human movement data into hardware-agnostic robot control across 6,015 unique tasks and 9,869 scenes.
- Through a Self Forcing distillation pipeline, DreamDojo achieves real-time performance at 10.81 FPS, enabling live teleoperation, long-horizon simulations, and model-based planning that boosts real-world robot success rates by 17%.