Interactive Teaching Algorithms for Inverse Reinforcement Learning
This addresses the challenge of inefficient learning in IRL for applications like robotics or autonomous systems, though it is incremental as it builds on existing IRL methods.
The paper tackles the problem of inverse reinforcement learning (IRL) by introducing a teacher to provide adaptive demonstrations, resulting in drastically sped-up learning progress in experiments with a car driving simulator.
We study the problem of inverse reinforcement learning (IRL) with the added twist that the learner is assisted by a helpful teacher. More formally, we tackle the following algorithmic question: How could a teacher provide an informative sequence of demonstrations to an IRL learner to speed up the learning process? We present an interactive teaching framework where a teacher adaptively chooses the next demonstration based on learner's current policy. In particular, we design teaching algorithms for two concrete settings: an omniscient setting where a teacher has full knowledge about the learner's dynamics and a blackbox setting where the teacher has minimal knowledge. Then, we study a sequential variant of the popular MCE-IRL learner and prove convergence guarantees of our teaching algorithm in the omniscient setting. Extensive experiments with a car driving simulator environment show that the learning progress can be speeded up drastically as compared to an uninformative teacher.