A Distributed Primal-Dual Method for Constrained Multi-agent Reinforcement Learning with General Parameterization
This work addresses the challenge of decentralized online learning under shared constraints in multi-agent systems, which is important for applications like resource allocation and robotics.
The paper proposes a fully decentralized primal-dual algorithm for constrained multi-agent reinforcement learning, proving convergence to an equilibrium and analyzing sub-optimality. It demonstrates the approach on a Cournot game test environment.
This paper proposes a novel distributed approach for solving a cooperative Constrained Multi-agent Reinforcement Learning (CMARL) problem, where agents seek to minimize a global objective function subject to shared constraints. Unlike existing methods that rely on centralized training or coordination, our approach enables fully decentralized online learning, with each agent maintaining local estimates of both primal and dual variables. Specifically, we develop a distributed primal-dual algorithm based on actor-critic methods, leveraging local information to estimate Lagrangian multipliers. We establish consensus among the Lagrangian multipliers across agents and prove the convergence of our algorithm to an equilibrium point, analyzing the sub-optimality of this equilibrium compared to the exact solution of the unparameterized problem. Furthermore, we introduce a constrained cooperative Cournot game with stochastic dynamics as a test environment to evaluate the algorithm's performance in complex, real-world scenarios.