Where Do Humans Look When Demonstrating to Robots? Human Gaze Behavior in Pick-and-Place Tasks Across Demonstration Devices
For researchers in imitation learning, this work highlights the critical impact of demonstration device choice on gaze behavior and downstream model performance, revealing a previously overlooked factor that can undermine the benefits of gaze-based approaches.
This paper investigates how different demonstration devices used in imitation learning affect human gaze behavior, finding that certain device properties shift gaze from task-goal cues to control-monitoring cues, which can degrade the performance of gaze-based imitation learning models below non-gaze baselines.
Imitation learning for generalizable performance often requires a large volume of demonstration data, making the process significantly costly. One promising strategy to address this challenge is to leverage the cognitive skills of human demonstrators with strong generalization capability, particularly by revealing the underlying task demands reflected in their gaze behavior. However, imitation learning typically involves humans collecting data using demonstration devices that emulate a robot's embodiment and visual condition. This raises the question of how such devices influence gaze behavior. We propose an experimental framework that systematically analyzes human demonstrators' gaze behavior across a spectrum of robot-emulating demonstration devices. Our experimental results show that certain device properties shift gaze from task-goal cues (e.g., objects) toward control-monitoring cues (e.g., the end-effector). Furthermore, these shifts directly affect the performance of typical gaze-based imitation learning models, sometimes degrading it below non-gaze baselines.