Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning
This work addresses safety constraints for multiagent systems in applications like energy and drones, offering incremental improvements to existing primal-dual methods.
The paper tackled the problem of enforcing safety constraints in multiagent reinforcement learning by analyzing the structural effects of penalty terms in primal-dual algorithms, showing that standard methods lead to weak safety but modifications enable probabilistic constraints and improved value estimation, resulting in a proposed algorithm that accelerates convergence to safe policies in simulations.
Constrained multiagent reinforcement learning (C-MARL) is gaining importance as MARL algorithms find new applications in real-world systems ranging from energy systems to drone swarms. Most C-MARL algorithms use a primal-dual approach to enforce constraints through a penalty function added to the reward. In this paper, we study the structural effects of this penalty term on the MARL problem. First, we show that the standard practice of using the constraint function as the penalty leads to a weak notion of safety. However, by making simple modifications to the penalty term, we can enforce meaningful probabilistic (chance and conditional value at risk) constraints. Second, we quantify the effect of the penalty term on the value function, uncovering an improved value estimation procedure. We use these insights to propose a constrained multiagent advantage actor critic (C-MAA2C) algorithm. Simulations in a simple constrained multiagent environment affirm that our reinterpretation of the primal-dual method in terms of probabilistic constraints is effective, and that our proposed value estimate accelerates convergence to a safe joint policy.