BC-IRL: Learning Generalizable Reward Functions from Demonstrations
This addresses a key limitation in IRL for robotics and AI by improving reward generalization, though it is an incremental advance over existing methods.
The paper tackles the problem of reward functions learned via inverse reinforcement learning (IRL) overfitting to demonstrations, which limits their generalization to new states. It introduces BC-IRL, a method that learns more generalizable reward functions, achieving over twice the success rate of baselines in robotic control tasks.
How well do reward functions learned with inverse reinforcement learning (IRL) generalize? We illustrate that state-of-the-art IRL algorithms, which maximize a maximum-entropy objective, learn rewards that overfit to the demonstrations. Such rewards struggle to provide meaningful rewards for states not covered by the demonstrations, a major detriment when using the reward to learn policies in new situations. We introduce BC-IRL a new inverse reinforcement learning method that learns reward functions that generalize better when compared to maximum-entropy IRL approaches. In contrast to the MaxEnt framework, which learns to maximize rewards around demonstrations, BC-IRL updates reward parameters such that the policy trained with the new reward matches the expert demonstrations better. We show that BC-IRL learns rewards that generalize better on an illustrative simple task and two continuous robotic control tasks, achieving over twice the success rate of baselines in challenging generalization settings.