Inverse Reinforcement Learning with the Average Reward Criterion
This work addresses a limitation in IRL for robotics and control tasks by removing the discount factor assumption, though it is incremental as it builds on existing IRL methods.
The paper tackles the problem of Inverse Reinforcement Learning (IRL) by proposing an average-reward framework to recover unknown policies and reward functions from expert samples, without assuming a known discount factor, and achieves complexity results of O(1/ε) for subproblems and O(1/ε²) for the main IRL problem.
We study the problem of Inverse Reinforcement Learning (IRL) with an average-reward criterion. The goal is to recover an unknown policy and a reward function when the agent only has samples of states and actions from an experienced agent. Previous IRL methods assume that the expert is trained in a discounted environment, and the discount factor is known. This work alleviates this assumption by proposing an average-reward framework with efficient learning algorithms. We develop novel stochastic first-order methods to solve the IRL problem under the average-reward setting, which requires solving an Average-reward Markov Decision Process (AMDP) as a subproblem. To solve the subproblem, we develop a Stochastic Policy Mirror Descent (SPMD) method under general state and action spaces that needs $\mathcal{O}(1/\varepsilon)$ steps of gradient computation. Equipped with SPMD, we propose the Inverse Policy Mirror Descent (IPMD) method for solving the IRL problem with a $\mathcal{O}(1/\varepsilon^2)$ complexity. To the best of our knowledge, the aforementioned complexity results are new in IRL. Finally, we corroborate our analysis with numerical experiments using the MuJoCo benchmark and additional control tasks.