A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guarantee
This work addresses the challenge of convergence guarantees in stochastic optimal control for researchers in control theory and reinforcement learning, but it appears incremental as it builds on existing policy gradient frameworks.
The authors tackled the problem of ensuring global convergence for policy gradient methods in continuous-time stochastic optimal control by analyzing the gradient flow as a limit of these methods, proving global convergence and establishing a convergence rate under regularity assumptions.
We consider policy gradient methods for stochastic optimal control problem in continuous time. In particular, we analyze the gradient flow for the control, viewed as a continuous time limit of the policy gradient method. We prove the global convergence of the gradient flow and establish a convergence rate under some regularity assumptions. The main novelty in the analysis is the notion of local optimal control function, which is introduced to characterize the local optimality of the iterate.