Simitate: A Hybrid Imitation Learning Benchmark
This work provides a standardized benchmark for imitation learning researchers, though it is incremental as it builds on existing benchmarking efforts.
The authors introduced Simitate, a hybrid benchmarking suite for evaluating imitation learning approaches, featuring a dataset of 1938 human activity sequences with RGB/depth streams and ground truth poses, along with a simulator and evaluation metrics.
We present Simitate --- a hybrid benchmarking suite targeting the evaluation of approaches for imitation learning. A dataset containing 1938 sequences where humans perform daily activities in a realistic environment is presented. The dataset is strongly coupled with an integration into a simulator. RGB and depth streams with a resolution of 960$\mathbb{\times}$540 at 30Hz and accurate ground truth poses for the demonstrator's hand, as well as the object in 6 DOF at 120Hz are provided. Along with our dataset we provide the 3D model of the used environment, labeled object images and pre-trained models. A benchmarking suite that aims at fostering comparability and reproducibility supports the development of imitation learning approaches. Further, we propose and integrate evaluation metrics on assessing the quality of effect and trajectory of the imitation performed in simulation. Simitate is available on our project website: \url{https://agas.uni-koblenz.de/data/simitate/}.