Convergence and Optimality of Policy Gradient Methods in Weakly Smooth Settings
This work addresses convergence analysis for policy gradient methods, offering more intuitive conditions that could benefit researchers and practitioners in reinforcement learning, though it appears incremental as it extends existing theory rather than introducing a new paradigm.
The authors tackled the problem of opaque and impractical convergence conditions for policy gradient methods in control and reinforcement learning, establishing explicit convergence rates for weakly smooth policy classes with L2 integrable gradient and providing performance guarantees for converged policies.
Policy gradient methods have been frequently applied to problems in control and reinforcement learning with great success, yet existing convergence analysis still relies on non-intuitive, impractical and often opaque conditions. In particular, existing rates are achieved in limited settings, under strict regularity conditions. In this work, we establish explicit convergence rates of policy gradient methods, extending the convergence regime to weakly smooth policy classes with $L_2$ integrable gradient. We provide intuitive examples to illustrate the insight behind these new conditions. Notably, our analysis also shows that convergence rates are achievable for both the standard policy gradient and the natural policy gradient algorithms under these assumptions. Lastly we provide performance guarantees for the converged policies.