A User's Guide to Calibrating Robotics Simulators
This work provides a standardized evaluation framework for sim-to-real algorithms, which is crucial for robotics researchers and practitioners to make informed decisions.
This paper addresses the lack of consistent testing and metrics for sim-to-real algorithms in robotics. It proposes a set of benchmarks and a framework to study these algorithms, conducting experiments across various simulated environments to characterize their performance.
Simulators are a critical component of modern robotics research. Strategies for both perception and decision making can be studied in simulation first before deployed to real world systems, saving on time and costs. Despite significant progress on the development of sim-to-real algorithms, the analysis of different methods is still conducted in an ad-hoc manner, without a consistent set of tests and metrics for comparison. This paper fills this gap and proposes a set of benchmarks and a framework for the study of various algorithms aimed to transfer models and policies learnt in simulation to the real world. We conduct experiments on a wide range of well known simulated environments to characterize and offer insights into the performance of different algorithms. Our analysis can be useful for practitioners working in this area and can help make informed choices about the behavior and main properties of sim-to-real algorithms. We open-source the benchmark, training data, and trained models, which can be found at https://github.com/NVlabs/sim-parameter-estimation.