The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audio
This work addresses the problem of enabling robots to learn from multimodal simulations for real-world tasks, representing a novel advancement in sim-to-real transfer beyond vision-only approaches.
The paper tackles the challenge of multimodal sim-to-real transfer for robots by introducing MultiGen, a framework that integrates generative models into physics simulators to synthesize realistic audio from simulation video, enabling training without real robot data. It demonstrates effective zero-shot transfer to real-world pouring tasks with novel containers and liquids.
Robots must integrate multiple sensory modalities to act effectively in the real world. Yet, learning such multimodal policies at scale remains challenging. Simulation offers a viable solution, but while vision has benefited from high-fidelity simulators, other modalities (e.g. sound) can be notoriously difficult to simulate. As a result, sim-to-real transfer has succeeded primarily in vision-based tasks, with multimodal transfer still largely unrealized. In this work, we tackle these challenges by introducing MultiGen, a framework that integrates large-scale generative models into traditional physics simulators, enabling multisensory simulation. We showcase our framework on the dynamic task of robot pouring, which inherently relies on multimodal feedback. By synthesizing realistic audio conditioned on simulation video, our method enables training on rich audiovisual trajectories -- without any real robot data. We demonstrate effective zero-shot transfer to real-world pouring with novel containers and liquids, highlighting the potential of generative modeling to both simulate hard-to-model modalities and close the multimodal sim-to-real gap.