CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning
This addresses a key challenge in offline IRL for robotics and control applications, though it is an incremental improvement over existing methods.
The paper tackles reward extrapolation error in offline inverse reinforcement learning by proposing CLARE, a conservative model-based algorithm that integrates conservatism into reward learning and uses an estimated dynamics model, achieving significant performance gains over state-of-the-art methods on MuJoCo tasks, especially with small datasets.
This work aims to tackle a major challenge in offline Inverse Reinforcement Learning (IRL), namely the reward extrapolation error, where the learned reward function may fail to explain the task correctly and misguide the agent in unseen environments due to the intrinsic covariate shift. Leveraging both expert data and lower-quality diverse data, we devise a principled algorithm (namely CLARE) that solves offline IRL efficiently via integrating "conservatism" into a learned reward function and utilizing an estimated dynamics model. Our theoretical analysis provides an upper bound on the return gap between the learned policy and the expert policy, based on which we characterize the impact of covariate shift by examining subtle two-tier tradeoffs between the exploitation (on both expert and diverse data) and exploration (on the estimated dynamics model). We show that CLARE can provably alleviate the reward extrapolation error by striking the right exploitation-exploration balance therein. Extensive experiments corroborate the significant performance gains of CLARE over existing state-of-the-art algorithms on MuJoCo continuous control tasks (especially with a small offline dataset), and the learned reward is highly instructive for further learning.