Back to Reality for Imitation Learning
This is a call to the robot learning community to develop tailored metrics for real-world robotics, addressing a foundational issue in evaluation.
The paper argues that imitation learning and robot learning should shift from data efficiency to time efficiency as the primary evaluation metric, better reflecting real-world human costs.
Imitation learning, and robot learning in general, emerged due to breakthroughs in machine learning, rather than breakthroughs in robotics. As such, evaluation metrics for robot learning are deeply rooted in those for machine learning, and focus primarily on data efficiency. We believe that a better metric for real-world robot learning is time efficiency, which better models the true cost to humans. This is a call to arms to the robot learning community to develop our own evaluation metrics, tailored towards the long-term goals of real-world robotics.