Solving Multi-Agent Safe Optimal Control with Distributed Epigraph Form MARL
This addresses safety and stability issues in multi-robot systems, but it is incremental as it builds on existing safe MARL methods with a novel optimization formulation.
The paper tackled the problem of unstable training in safe multi-agent reinforcement learning for constrained Markov decision processes by proposing a distributed epigraph form approach, resulting in the Def-MARL algorithm that achieved the best overall performance, satisfied safety constraints, and maintained stable training across 8 simulation tasks and real-world quadcopter experiments.
Tasks for multi-robot systems often require the robots to collaborate and complete a team goal while maintaining safety. This problem is usually formalized as a constrained Markov decision process (CMDP), which targets minimizing a global cost and bringing the mean of constraint violation below a user-defined threshold. Inspired by real-world robotic applications, we define safety as zero constraint violation. While many safe multi-agent reinforcement learning (MARL) algorithms have been proposed to solve CMDPs, these algorithms suffer from unstable training in this setting. To tackle this, we use the epigraph form for constrained optimization to improve training stability and prove that the centralized epigraph form problem can be solved in a distributed fashion by each agent. This results in a novel centralized training distributed execution MARL algorithm named Def-MARL. Simulation experiments on 8 different tasks across 2 different simulators show that Def-MARL achieves the best overall performance, satisfies safety constraints, and maintains stable training. Real-world hardware experiments on Crazyflie quadcopters demonstrate the ability of Def-MARL to safely coordinate agents to complete complex collaborative tasks compared to other methods.