Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm
This addresses stability issues in IRL for robotics and behavior modeling, offering a non-adversarial method with formal guarantees, though it builds incrementally on existing trust region concepts.
The paper tackles the instability in Inverse Reinforcement Learning (IRL) by introducing Trust Region Reward Optimization (TRRO), a framework that guarantees monotonic improvement in likelihood of expert behavior, leading to the Proximal Inverse Reward Optimization (PIRO) algorithm, which matches or surpasses state-of-the-art baselines in reward recovery and policy imitation on benchmarks like MuJoCo and Gym-Robotics.
Inverse Reinforcement Learning (IRL) learns a reward function to explain expert demonstrations. Modern IRL methods often use the adversarial (minimax) formulation that alternates between reward and policy optimization, which often lead to unstable training. Recent non-adversarial IRL approaches improve stability by jointly learning reward and policy via energy-based formulations but lack formal guarantees. This work bridges this gap. We first present a unified view showing canonical non-adversarial methods explicitly or implicitly maximize the likelihood of expert behavior, which is equivalent to minimizing the expected return gap. This insight leads to our main contribution: Trust Region Reward Optimization (TRRO), a framework that guarantees monotonic improvement in this likelihood via a Minorization-Maximization process. We instantiate TRRO into Proximal Inverse Reward Optimization (PIRO), a practical and stable IRL algorithm. Theoretically, TRRO provides the IRL counterpart to the stability guarantees of Trust Region Policy Optimization (TRPO) in forward RL. Empirically, PIRO matches or surpasses state-of-the-art baselines in reward recovery, policy imitation with high sample efficiency on MuJoCo and Gym-Robotics benchmarks and a real-world animal behavior modeling task.