f-IRL: Inverse Reinforcement Learning via State Marginal Matching
This addresses the challenge of specifying costs or programming behaviors in robotic tasks by providing a more efficient and effective inverse reinforcement learning approach, though it is incremental as it builds on existing state marginal matching and f-divergence concepts.
The paper tackles the problem of learning reward functions from expert state densities in imitation learning, proposing f-IRL which uses analytic gradients of f-divergences to recover stationary rewards via gradient descent. The method outperforms adversarial imitation learning in sample efficiency and expert trajectory requirements on benchmarks, and the recovered rewards enable quick solving of downstream tasks like hard-to-explore problems and behavior transfer.
Imitation learning is well-suited for robotic tasks where it is difficult to directly program the behavior or specify a cost for optimal control. In this work, we propose a method for learning the reward function (and the corresponding policy) to match the expert state density. Our main result is the analytic gradient of any f-divergence between the agent and expert state distribution w.r.t. reward parameters. Based on the derived gradient, we present an algorithm, f-IRL, that recovers a stationary reward function from the expert density by gradient descent. We show that f-IRL can learn behaviors from a hand-designed target state density or implicitly through expert observations. Our method outperforms adversarial imitation learning methods in terms of sample efficiency and the required number of expert trajectories on IRL benchmarks. Moreover, we show that the recovered reward function can be used to quickly solve downstream tasks, and empirically demonstrate its utility on hard-to-explore tasks and for behavior transfer across changes in dynamics.