Vairiational Stochastic Games
This work provides a novel variational inference approach for decentralized multi-agent systems, which is an incremental advancement in multi-agent reinforcement learning.
The paper tackles the challenge of extending the Control as Inference framework to decentralized multi-agent stochastic games, addressing non-stationarity and unaligned objectives, and proves that the resulting policies form an ε-Nash equilibrium with theoretical convergence guarantees.
The Control as Inference (CAI) framework has successfully transformed single-agent reinforcement learning (RL) by reframing control tasks as probabilistic inference problems. However, the extension of CAI to multi-agent, general-sum stochastic games (SGs) remains underexplored, particularly in decentralized settings where agents operate independently without centralized coordination. In this paper, we propose a novel variational inference framework tailored to decentralized multi-agent systems. Our framework addresses the challenges posed by non-stationarity and unaligned agent objectives, proving that the resulting policies form an $ε$-Nash equilibrium. Additionally, we demonstrate theoretical convergence guarantees for the proposed decentralized algorithms. Leveraging this framework, we instantiate multiple algorithms to solve for Nash equilibrium, mean-field Nash equilibrium, and correlated equilibrium, with rigorous theoretical convergence analysis.