Convergence Rates for Localized Actor-Critic in Networked Markov Potential Games
This addresses scalability issues for researchers and practitioners in multi-agent systems by providing a method that overcomes the curse of dimensionality, though it is incremental as it builds on existing actor-critic frameworks.
The paper tackles the problem of scalability in multi-agent competitive games by proposing a localized actor-critic algorithm for networked Markov potential games, achieving an $ ilde{\mathcal{O}}( ildeε^{-4})$ sample complexity for averaged Nash regret that does not depend on the number of agents.
We introduce a class of networked Markov potential games in which agents are associated with nodes in a network. Each agent has its own local potential function, and the reward of each agent depends only on the states and actions of the agents within a neighborhood. In this context, we propose a localized actor-critic algorithm. The algorithm is scalable since each agent uses only local information and does not need access to the global state. Further, the algorithm overcomes the curse of dimensionality through the use of function approximation. Our main results provide finite-sample guarantees up to a localization error and a function approximation error. Specifically, we achieve an $\tilde{\mathcal{O}}(\tildeε^{-4})$ sample complexity measured by the averaged Nash regret. This is the first finite-sample bound for multi-agent competitive games that does not depend on the number of agents.