Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games
This provides a theoretical guarantee for policy gradient methods in multi-agent reinforcement learning, addressing convergence issues in stochastic environments, though it is incremental as it builds on existing linear-quadratic game frameworks.
The paper tackles the problem of finding Nash equilibria in N-player general-sum linear-quadratic games with stochastic dynamics, proving that the natural policy gradient method converges globally to the Nash equilibrium under a noise condition, and demonstrates through numerical experiments that noise enables convergence where deterministic settings fail.
We consider a general-sum N-player linear-quadratic game with stochastic dynamics over a finite horizon and prove the global convergence of the natural policy gradient method to the Nash equilibrium. In order to prove the convergence of the method, we require a certain amount of noise in the system. We give a condition, essentially a lower bound on the covariance of the noise in terms of the model parameters, in order to guarantee convergence. We illustrate our results with numerical experiments to show that even in situations where the policy gradient method may not converge in the deterministic setting, the addition of noise leads to convergence.