Fast Rates for Inverse Reinforcement Learning
This work provides the first fast statistical rates for inverse reinforcement learning, addressing a key theoretical gap for practitioners needing sample-efficient reward learning.
The paper establishes that for entropy-regularized min-max inverse reinforcement learning with linear reward classes, both the trajectory-level KL divergence and squared parameter error in the Hessian norm decay at the fast rate O(n^{-1}), where n is the number of expert trajectories, under misspecification and without exploration assumptions.
We establish novel structural and statistical results for entropy-regularized min-max inverse reinforcement learning (Min-Max-IRL) with linear reward classes in finite-horizon MDPs with Borel state and action spaces. On the structural side, we show that maximum likelihood estimation (MLE) and Min-Max-IRL are equivalent at the population level, and at the empirical level under deterministic dynamics. On the statistical side, exploiting pseudo-self-concordance of the Min-Max-IRL loss, we prove that both the trajectory-level KL divergence and the squared parameter error in the Hessian norm decay at the fast rate $\mathcal{O}(n^{-1})$, where $n$ is the number of expert trajectories. Our guarantees apply under misspecification and require no exploration assumptions. We further extend reward-identifiability results to general Borel spaces and derive novel results on the derivatives of the soft-optimal value function with respect to reward parameters.