A Framework and Method for Online Inverse Reinforcement Learning
This addresses the need for efficient IRL in applications where observations are accrued incrementally, offering improved performance and speed-up in simulated robotic tasks.
The paper tackles the problem of online inverse reinforcement learning (IRL) by introducing the first formal framework called incremental IRL (I2RL) and a new method based on maximum entropy IRL with hidden variables, showing monotonically improving performance with more data and probabilistically bounded error in experiments.
Inverse reinforcement learning (IRL) is the problem of learning the preferences of an agent from the observations of its behavior on a task. While this problem has been well investigated, the related problem of {\em online} IRL---where the observations are incrementally accrued, yet the demands of the application often prohibit a full rerun of an IRL method---has received relatively less attention. We introduce the first formal framework for online IRL, called incremental IRL (I2RL), and a new method that advances maximum entropy IRL with hidden variables, to this setting. Our formal analysis shows that the new method has a monotonically improving performance with more demonstration data, as well as probabilistically bounded error, both under full and partial observability. Experiments in a simulated robotic application of penetrating a continuous patrol under occlusion shows the relatively improved performance and speed up of the new method and validates the utility of online IRL.