Global Convergence of Policy Gradient Methods in Reinforcement Learning, Games and Control
This work provides a synthesis of incremental advances in convergence analysis for policy gradient methods, relevant for researchers in reinforcement learning, games, and control.
The paper addresses the challenge of ensuring global optimality in policy gradient methods for sequential decision-making, highlighting recent progress in developing methods with global convergence guarantees and finite-time convergence rates.
Policy gradient methods, where one searches for the policy of interest by maximizing the value functions using first-order information, become increasingly popular for sequential decision making in reinforcement learning, games, and control. Guaranteeing the global optimality of policy gradient methods, however, is highly nontrivial due to nonconcavity of the value functions. In this exposition, we highlight recent progresses in understanding and developing policy gradient methods with global convergence guarantees, putting an emphasis on their finite-time convergence rates with regard to salient problem parameters.