Convergence of Value Aggregation for Imitation Learning
This addresses theoretical gaps for researchers in imitation learning, offering insights to stabilize algorithms and improve practical implementation.
The paper tackles the problem of convergence and performance guarantees in value aggregation for imitation learning, showing that the policy sequence does not always converge and providing a stability condition and non-asymptotic bound for the last policy's performance.
Value aggregation is a general framework for solving imitation learning problems. Based on the idea of data aggregation, it generates a policy sequence by iteratively interleaving policy optimization and evaluation in an online learning setting. While the existence of a good policy in the policy sequence can be guaranteed non-asymptotically, little is known about the convergence of the sequence or the performance of the last policy. In this paper, we debunk the common belief that value aggregation always produces a convergent policy sequence with improving performance. Moreover, we identify a critical stability condition for convergence and provide a tight non-asymptotic bound on the performance of the last policy. These new theoretical insights let us stabilize problems with regularization, which removes the inconvenient process of identifying the best policy in the policy sequence in stochastic problems.