Inverse Reinforcement Learning by Estimating Expertise of Demonstrators
This addresses the problem of handling varied real-world data for imitation learning practitioners, though it is incremental as it builds on existing inverse reinforcement learning methods.
The paper tackles the challenge of learning from suboptimal and heterogeneous demonstrations in imitation learning by introducing IRLEED, a framework that estimates demonstrator expertise without prior knowledge, resulting in improved adaptability and effectiveness in both online and offline settings.
In Imitation Learning (IL), utilizing suboptimal and heterogeneous demonstrations presents a substantial challenge due to the varied nature of real-world data. However, standard IL algorithms consider these datasets as homogeneous, thereby inheriting the deficiencies of suboptimal demonstrators. Previous approaches to this issue rely on impractical assumptions like high-quality data subsets, confidence rankings, or explicit environmental knowledge. This paper introduces IRLEED, Inverse Reinforcement Learning by Estimating Expertise of Demonstrators, a novel framework that overcomes these hurdles without prior knowledge of demonstrator expertise. IRLEED enhances existing Inverse Reinforcement Learning (IRL) algorithms by combining a general model for demonstrator suboptimality to address reward bias and action variance, with a Maximum Entropy IRL framework to efficiently derive the optimal policy from diverse, suboptimal demonstrations. Experiments in both online and offline IL settings, with simulated and human-generated data, demonstrate IRLEED's adaptability and effectiveness, making it a versatile solution for learning from suboptimal demonstrations.