Learning Transferable Dynamics Priors from Action to World Modeling
For robot learning researchers, this work demonstrates that action-conditioned world model pretraining provides reusable dynamics priors that benefit both simulation and policy learning, though the gains are incremental over existing video prediction and policy learning methods.
The paper introduces A2World, a multi-view interactive diffusion world model pretrained on large-scale robot manipulation data with real action annotations, and shows that the learned dynamics priors transfer to both simulator-based policy evaluation and action prediction tasks, improving performance in simulation and real-robot settings.
We study action-conditioned world modeling as a scalable way to learn transferable dynamics priors for robot learning. By pretraining a model to predict how actions drive visual scene evolution, the resulting world model captures reusable interaction dynamics beyond appearance-level video generation. Concretely, we pretrain a multi-view interactive base diffusion world model, A2World, on large-scale robot manipulation data with real action annotations. We validate the learned dynamics priors from two complementary perspectives. First, we adapt A2World into a task- or scene-specialized real-world simulator, A2World-sim, whose long-horizon rollouts support simulator-based policy evaluation and scalable what-if analysis by replacing real-robot rollouts with world model rollouts. Second, starting from the same pretrained weights, we adapt A2World into a video-action joint prediction model, A2World-policy, that predicts actions under visual and instruction conditioning. Experiments across simulation benchmarks and real-robot settings demonstrate that action-conditioned world model pretraining yields transferable dynamics priors that benefit both simulator-centric and policy-centric robot learning.