Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch
This addresses a practical issue in IRL for robotics or simulation domains where dynamics differ, but it is incremental as it builds on existing MCE IRL methods.
The paper tackles the problem of inverse reinforcement learning (IRL) when there is a mismatch in transition dynamics between expert and learner, providing a tight performance bound and proposing a robust algorithm that shows stable performance in experiments.
We study the inverse reinforcement learning (IRL) problem under a transition dynamics mismatch between the expert and the learner. Specifically, we consider the Maximum Causal Entropy (MCE) IRL learner model and provide a tight upper bound on the learner's performance degradation based on the $\ell_1$-distance between the transition dynamics of the expert and the learner. Leveraging insights from the Robust RL literature, we propose a robust MCE IRL algorithm, which is a principled approach to help with this mismatch. Finally, we empirically demonstrate the stable performance of our algorithm compared to the standard MCE IRL algorithm under transition dynamics mismatches in both finite and continuous MDP problems.