Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning
This addresses coordination challenges in MARL for domains like gaming and traffic control, offering a method to learn tradeoffs between centralization and decentralization, though it is incremental as it builds on existing coordination graph and actor-critic approaches.
The paper tackles the problem of multi-agent reinforcement learning requiring coordination in large joint action spaces by introducing the deep implicit coordination graph (DICG) architecture, which learns dynamic coordination graphs and uses graph neural networks to improve coordination, resulting in solving relative overgeneralization in predatory-prey tasks and outperforming baselines on SMAC and traffic junction environments.
Multi-agent reinforcement learning (MARL) requires coordination to efficiently solve certain tasks. Fully centralized control is often infeasible in such domains due to the size of joint action spaces. Coordination graph based formalization allows reasoning about the joint action based on the structure of interactions. However, they often require domain expertise in their design. This paper introduces the deep implicit coordination graph (DICG) architecture for such scenarios. DICG consists of a module for inferring the dynamic coordination graph structure which is then used by a graph neural network based module to learn to implicitly reason about the joint actions or values. DICG allows learning the tradeoff between full centralization and decentralization via standard actor-critic methods to significantly improve coordination for domains with large number of agents. We apply DICG to both centralized-training-centralized-execution and centralized-training-decentralized-execution regimes. We demonstrate that DICG solves the relative overgeneralization pathology in predatory-prey tasks as well as outperforms various MARL baselines on the challenging StarCraft II Multi-agent Challenge (SMAC) and traffic junction environments.