RoDyn: Taming Interactive Robot-Dynamic 2.5D World Model for Robotic Manipulation
For robotic manipulation tasks requiring spatial and kinematic reasoning, RoDyn provides a novel world model that significantly improves downstream policy learning, addressing a key limitation of 2D video-based models.
RoDyn introduces a geometry-aware 2.5D world model for robotic manipulation that couples visual and spatial priors via a Robot-Dynamic Tokenizer and Mask-guided Autoregressive architecture, achieving state-of-the-art generation fidelity and a 42% improvement in real-world imitation learning success rates over 2D baselines.
Learned world models hold significant potential as neural simulators for robotic manipulation. However, prevalent 2D video-based models inherently lack the spatial and kinematic reasoning crucial for physical interactions. We introduce RoDyn, a novel Robot-Dynamic 2.5D World Model that formulates environmental dynamics within a highly efficient, geometry-aware latent space. Through the proposed Robot-Dynamic Tokenizer, we explicitly couple semantic visual appearances with spatial and agent-centric priors via an RGB-dominated cross-attention mechanism and dynamic mask guidance. Furthermore, by injecting these mask priors directly into sequence transitions, our Mask-guided Autoregressive architecture drives the model to focus on active robot-object interaction regions. Extensive experiments demonstrate that RoDyn establishes SOTA generation fidelity across large-scale datasets. Crucially, it translates these predictive capabilities into substantial downstream gains, accelerating model-based reinforcement learning and achieving a 42\% improvement in real-world imitation learning success rates over pure 2D baselines.