Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime
This work addresses policy optimization challenges in reinforcement learning for researchers, providing theoretical guarantees but is incremental as it builds on existing mean-field and gradient flow frameworks.
The authors tackled the problem of policy optimization for infinite-horizon discounted Markov Decision Processes using softmax policy and nonlinear function approximation, proving global optimality of fixed points under mild initialization conditions in the mean-field regime.
We study the problem of policy optimization for infinite-horizon discounted Markov Decision Processes with softmax policy and nonlinear function approximation trained with policy gradient algorithms. We concentrate on the training dynamics in the mean-field regime, modeling e.g., the behavior of wide single hidden layer neural networks, when exploration is encouraged through entropy regularization. The dynamics of these models is established as a Wasserstein gradient flow of distributions in parameter space. We further prove global optimality of the fixed points of this dynamics under mild conditions on their initialization.