Bridging the Imitation Gap by Adaptive Insubordination
This addresses a specific problem in imitation learning for tasks requiring frequent switches between exploration and memorization, offering an incremental improvement over prior methods.
The paper tackles the imitation gap in imitation learning when a teaching agent uses privileged information unavailable to the student, leading to poor performance, and proposes Adaptive Insubordination (ADVISOR) to dynamically switch between imitation and exploration, outperforming baseline methods on challenging tasks.
In practice, imitation learning is preferred over pure reinforcement learning whenever it is possible to design a teaching agent to provide expert supervision. However, we show that when the teaching agent makes decisions with access to privileged information that is unavailable to the student, this information is marginalized during imitation learning, resulting in an "imitation gap" and, potentially, poor results. Prior work bridges this gap via a progression from imitation learning to reinforcement learning. While often successful, gradual progression fails for tasks that require frequent switches between exploration and memorization. To better address these tasks and alleviate the imitation gap we propose 'Adaptive Insubordination' (ADVISOR). ADVISOR dynamically weights imitation and reward-based reinforcement learning losses during training, enabling on-the-fly switching between imitation and exploration. On a suite of challenging tasks set within gridworlds, multi-agent particle environments, and high-fidelity 3D simulators, we show that on-the-fly switching with ADVISOR outperforms pure imitation, pure reinforcement learning, as well as their sequential and parallel combinations.