Model-Free Imitation Learning with Policy Optimization
This work addresses scalability issues in imitation learning for robotics or AI agents, though it is incremental as it builds on existing apprenticeship learning and policy optimization methods.
The paper tackles the problem of imitation learning in large, high-dimensional environments by developing model-free algorithms based on policy gradients, achieving guaranteed convergence to local minima without requiring optimal planning.
In imitation learning, an agent learns how to behave in an environment with an unknown cost function by mimicking expert demonstrations. Existing imitation learning algorithms typically involve solving a sequence of planning or reinforcement learning problems. Such algorithms are therefore not directly applicable to large, high-dimensional environments, and their performance can significantly degrade if the planning problems are not solved to optimality. Under the apprenticeship learning formalism, we develop alternative model-free algorithms for finding a parameterized stochastic policy that performs at least as well as an expert policy on an unknown cost function, based on sample trajectories from the expert. Our approach, based on policy gradients, scales to large continuous environments with guaranteed convergence to local minima.