Near-Optimal Policy Optimization for Correlated Equilibrium in General-Sum Markov Games
This work provides a significant improvement in convergence rates for correlated equilibrium computation in game theory, impacting multi-agent reinforcement learning and strategic decision-making.
The paper tackles the problem of computing correlated equilibria in multi-player general-sum Markov Games, achieving a near-optimal convergence rate of $ ilde{O}(T^{-1})$ with an uncoupled policy optimization algorithm.
We study policy optimization algorithms for computing correlated equilibria in multi-player general-sum Markov Games. Previous results achieve $O(T^{-1/2})$ convergence rate to a correlated equilibrium and an accelerated $O(T^{-3/4})$ convergence rate to the weaker notion of coarse correlated equilibrium. In this paper, we improve both results significantly by providing an uncoupled policy optimization algorithm that attains a near-optimal $\tilde{O}(T^{-1})$ convergence rate for computing a correlated equilibrium. Our algorithm is constructed by combining two main elements (i) smooth value updates and (ii) the optimistic-follow-the-regularized-leader algorithm with the log barrier regularizer.