Neural Motion Simulator: Pushing the Limit of World Models in Reinforcement Learning
This work addresses the challenge of sample efficiency and generalization in reinforcement learning for embodied systems, representing an incremental advancement by improving world modeling capabilities.
The authors tackled the problem of modeling motion dynamics for embodied systems in reinforcement learning by introducing the neural motion simulator (MoSim), which achieves state-of-the-art performance in physical state prediction and enables efficient skill acquisition and zero-shot reinforcement learning.
An embodied system must not only model the patterns of the external world but also understand its own motion dynamics. A motion dynamic model is essential for efficient skill acquisition and effective planning. In this work, we introduce the neural motion simulator (MoSim), a world model that predicts the future physical state of an embodied system based on current observations and actions. MoSim achieves state-of-the-art performance in physical state prediction and provides competitive performance across a range of downstream tasks. This works shows that when a world model is accurate enough and performs precise long-horizon predictions, it can facilitate efficient skill acquisition in imagined worlds and even enable zero-shot reinforcement learning. Furthermore, MoSim can transform any model-free reinforcement learning (RL) algorithm into a model-based approach, effectively decoupling physical environment modeling from RL algorithm development. This separation allows for independent advancements in RL algorithms and world modeling, significantly improving sample efficiency and enhancing generalization capabilities. Our findings highlight that world models for motion dynamics is a promising direction for developing more versatile and capable embodied systems.