On the Global Convergence of Imitation Learning: A Case for Linear Quadratic Regulator
This work addresses instability in imitation learning, with potential applications in reinforcement learning and generative adversarial learning, though it is incremental as a step towards broader understanding.
The paper tackles the global convergence of generative adversarial imitation learning for linear quadratic regulators by analyzing the alternating gradient algorithm, establishing a Q-linear convergence rate to a unique saddle point that recovers the optimal policy and reward function.
We study the global convergence of generative adversarial imitation learning for linear quadratic regulators, which is posed as minimax optimization. To address the challenges arising from non-convex-concave geometry, we analyze the alternating gradient algorithm and establish its Q-linear rate of convergence to a unique saddle point, which simultaneously recovers the globally optimal policy and reward function. We hope our results may serve as a small step towards understanding and taming the instability in imitation learning as well as in more general non-convex-concave alternating minimax optimization that arises from reinforcement learning and generative adversarial learning.