GPUSimBench: Towards Scalable and Reliable GPU-Accelerated Simulators in Embodied AI
For embodied AI researchers relying on GPU simulators, this work identifies critical trade-offs between scalability and reproducibility that must be addressed for reliable robot learning.
GPUSimBench exposes hidden limits of GPU-accelerated simulators (Isaac Lab, Genesis) for embodied AI, revealing that unbounded scaling introduces non-determinism and compromises reproducibility, with significant run-to-run variability even under identical conditions.
Data-driven embodied AI is rapidly transitioning into a paradigm that scales training through massively parallel simulation, where GPU-accelerated simulators serve as the foundational data infrastructure. However, as computational throughput scales, the underlying trade-offs between parallel efficiency, physical fidelity, and execution determinism remain largely unexamined, hindering the development of reliable robot learning. In this paper, we expose the hidden limits of mainstream GPU-based robotic simulators (e.g., Isaac Lab, Genesis) by introducing GPUSimBench, which focuses on scalability, physical consistency, and computational determinism. First, GPUSimBench establishes a physical grounding evaluation with a controlled inclined-plane task, quantifying the distributional alignment between simulated dynamics and their real-world counterparts. Second, we benchmark parallel scalability by measuring throughput and memory footprints across scaling environment counts. Crucially, beyond standard performance metrics, we unveil and quantify the inherent non-determinism introduced by GPU-batched execution, characterized by significant run-to-run and inter-environment variability even under identical initial conditions. Finally, we identify four empirical regimes of stochasticity within current simulator stacks, highlighting that unbounded scaling can compromise reproducibility without explicit constraints.