GaussGym: An open-source real-to-sim framework for learning locomotion from pixels
This work addresses the bottleneck of scalable and realistic simulation for robot learning, enabling faster training with diverse, photorealistic environments, though it is incremental as it builds on existing simulation and rendering techniques.
The authors tackled the problem of slow and low-fidelity robot simulation by integrating 3D Gaussian Splatting into physics simulators, achieving over 100,000 steps per second on consumer GPUs while maintaining high visual fidelity, and demonstrated its use in sim-to-real robotics with improved navigation through visual semantics.
We present a novel approach for photorealistic robot simulation that integrates 3D Gaussian Splatting as a drop-in renderer within vectorized physics simulators such as IsaacGym. This enables unprecedented speed -- exceeding 100,000 steps per second on consumer GPUs -- while maintaining high visual fidelity, which we showcase across diverse tasks. We additionally demonstrate its applicability in a sim-to-real robotics setting. Beyond depth-based sensing, our results highlight how rich visual semantics improve navigation and decision-making, such as avoiding undesirable regions. We further showcase the ease of incorporating thousands of environments from iPhone scans, large-scale scene datasets (e.g., GrandTour, ARKit), and outputs from generative video models like Veo, enabling rapid creation of realistic training worlds. This work bridges high-throughput simulation and high-fidelity perception, advancing scalable and generalizable robot learning. All code and data will be open-sourced for the community to build upon. Videos, code, and data available at https://escontrela.me/gauss_gym/.