On the Effective Horizon of Inverse Reinforcement Learning
This work addresses a practical bottleneck in IRL for researchers and practitioners by providing a principled method to choose the horizon, though it is incremental as it builds on existing IRL formulations.
The paper tackles the problem of selecting the time horizon in inverse reinforcement learning (IRL), showing that a shorter effective horizon than the ground-truth can improve accuracy and computational efficiency by reducing overfitting with limited data.
Inverse reinforcement learning (IRL) algorithms often rely on (forward) reinforcement learning or planning, over a given time horizon, to compute an approximately optimal policy for a hypothesized reward function; they then match this policy with expert demonstrations. The time horizon plays a critical role in determining both the accuracy of reward estimates and the computational efficiency of IRL algorithms. Interestingly, an *effective time horizon* shorter than the ground-truth value often produces better results faster. This work formally analyzes this phenomenon and provides an explanation: the time horizon controls the complexity of an induced policy class and mitigates overfitting with limited data. This analysis provides a guide for the principled choice of the effective horizon for IRL. It also prompts us to re-examine the classic IRL formulation: it is more natural to learn jointly the reward and the effective horizon rather than the reward alone with a given horizon. To validate our findings, we implement a cross-validation extension and the experimental results support the theoretical analysis. The project page and code are publicly available.