Decentralized MARL for Coarse Correlated Equilibrium in Aggregative Markov Games
For multi-agent reinforcement learning, this work provides a decentralized, model-free algorithm that leverages aggregative structure to achieve efficient learning, addressing a gap in existing CCE learning methods.
This paper proposes a decentralized V-learning algorithm for aggregative Markov games that achieves an epsilon-approximate Coarse Correlated Equilibrium in O(S Amax T^5 / epsilon^2) episodes, avoiding the curse of multiagents. Numerical results verify the theoretical findings.
This paper studies the problem of decentralized learning of Coarse Correlated Equilibrium (CCE) in aggregative Markov games (AMGs), where each agent's instantaneous reward depends only on its own action and an aggregate quantity. Existing CCE learning algorithms for general Markov games are not designed to leverage the aggregative structure, and research on decentralized CCE learning for AMGs remains limited. We propose an adaptive stage-based V-learning algorithm that exploits the aggregative structure under a fully decentralized information setting. Based on the two-timescale idea, the algorithm partitions learning into stages and adjusts stage lengths based on the variability of aggregate signals, while using no-regret updates within each stage. We prove the algorithm achieves an epsilon-approximate CCE in O(S Amax T5 / epsilon2) episodes, avoiding the curse of multiagents which commonly arises in MARL. Numerical results verify the theoretical findings, and the decentralized, model-free design enables easy extension to large-scale multi-agent scenarios.