Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors
For robotic manipulation, this work shows that world-action models can be trained from synthetic priors and deployed zero-shot, reducing the need for expensive real-world demonstrations.
This paper demonstrates the first successful sim-to-real transfer of a world-action model for robotic manipulation, achieving a 35% average success rate zero-shot on a Franka Robot across three tasks using only synthetic demonstrations.
Bridging the sim-to-real gap is a core challenge in deploying learned manipulation policies. Sim-to-real learning is attractive because it can replace expensive real robot demonstrations with scalable synthetic data, yet world-action models have not previously been shown to transfer from simulation to real robotic manipulation. We study whether a world-action model can be trained from synthetic priors and deployed zero-shot in the real world. To this end, we build upon Cosmos Policy, a video diffusion model adapted for visuomotor control. We construct simulation environments with extensive domain randomization and generate demonstrations using the AnyTask motion planning pipeline. We evaluate our approach across object lifting, drawer opening, and pick-and-place tasks using ${\sim}800$ synthetic demonstrations per task and no real demonstrations. When deployed zero-shot on a Franka Robot, our policy attains a 35\% average success rate. To our knowledge, this represents the first successful sim-to-real transfer of a world-action model for robotic manipulation.