Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance
This work provides empirical insights and guidelines for researchers and practitioners in robotics to inform method selection and deployment, though it is incremental as it benchmarks existing methods without introducing new techniques.
The authors conducted a large-scale benchmark study evaluating four motion-based learning from demonstration approaches across manipulation tasks with varying complexities, user expertise, and starting conditions, training 180 models and testing 720 reproductions on a physical robot to detail performance impacts.
In this work, we contribute a large-scale study benchmarking the performance of multiple motion-based learning from demonstration approaches. Given the number and diversity of existing methods, it is critical that comprehensive empirical studies be performed comparing the relative strengths of these learning techniques. In particular, we evaluate four different approaches based on properties an end user may desire for real-world tasks. To perform this evaluation, we collected data from nine participants, across four different manipulation tasks with varying starting conditions. The resulting demonstrations were used to train 180 task models and evaluated on 720 task reproductions on a physical robot. Our results detail how i) complexity of the task, ii) the expertise of the human demonstrator, and iii) the starting configuration of the robot affect task performance. The collected dataset of demonstrations, robot executions, and evaluations are being made publicly available. Research insights and guidelines are also provided to guide future research and deployment choices about these approaches.