Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
This addresses the problem of reducing human workload in robot learning for continuous and discrete control tasks, representing an incremental improvement over existing methods.
The paper tackles the high cognitive demands on human supervisors in Interactive Imitation Learning by proposing the Adaptive Intervention Mechanism (AIM), a robot-gated algorithm that reduces expert monitoring efforts by 40% in human take-over cost and learning efficiency compared to a baseline.
Interactive Imitation Learning (IIL) allows agents to acquire desired behaviors through human interventions, but current methods impose high cognitive demands on human supervisors. We propose the Adaptive Intervention Mechanism (AIM), a novel robot-gated IIL algorithm that learns an adaptive criterion for requesting human demonstrations. AIM utilizes a proxy Q-function to mimic the human intervention rule and adjusts intervention requests based on the alignment between agent and human actions. By assigning high Q-values when the agent deviates from the expert and decreasing these values as the agent becomes proficient, the proxy Q-function enables the agent to assess the real-time alignment with the expert and request assistance when needed. Our expert-in-the-loop experiments reveal that AIM significantly reduces expert monitoring efforts in both continuous and discrete control tasks. Compared to the uncertainty-based baseline Thrifty-DAgger, our method achieves a 40% improvement in terms of human take-over cost and learning efficiency. Furthermore, AIM effectively identifies safety-critical states for expert assistance, thereby collecting higher-quality expert demonstrations and reducing overall expert data and environment interactions needed. Code and demo video are available at https://github.com/metadriverse/AIM.