A Novel Variational Lower Bound for Inverse Reinforcement Learning
This addresses the problem of reward engineering in IRL for researchers and practitioners, but it appears incremental as it builds on existing probabilistic graphical models and lower bound methods.
The paper tackles the challenge of inverse reinforcement learning (IRL) in large, high-dimensional problems with unknown dynamics by introducing a new Variational Lower Bound for IRL (VLB-IRL), which learns a reward function and policy simultaneously to achieve expert-level performance and outperforms state-of-the-art IRL algorithms on several domains.
Inverse reinforcement learning (IRL) seeks to learn the reward function from expert trajectories, to understand the task for imitation or collaboration thereby removing the need for manual reward engineering. However, IRL in the context of large, high-dimensional problems with unknown dynamics has been particularly challenging. In this paper, we present a new Variational Lower Bound for IRL (VLB-IRL), which is derived under the framework of a probabilistic graphical model with an optimality node. Our method simultaneously learns the reward function and policy under the learned reward function by maximizing the lower bound, which is equivalent to minimizing the reverse Kullback-Leibler divergence between an approximated distribution of optimality given the reward function and the true distribution of optimality given trajectories. This leads to a new IRL method that learns a valid reward function such that the policy under the learned reward achieves expert-level performance on several known domains. Importantly, the method outperforms the existing state-of-the-art IRL algorithms on these domains by demonstrating better reward from the learned policy.