Option-Critic in Cooperative Multi-agent Systems
This addresses the problem of efficient planning and learning in decentralized multi-agent systems for researchers in AI and robotics, but it is incremental as it builds on existing options and multi-agent frameworks.
The paper tackles learning temporal abstractions in cooperative multi-agent systems by proposing the Distributed Option Critic (DOC) algorithm, which uses centralized evaluation and decentralized improvement, and it empirically shows competitive performance and scalability with the number of agents.
In this paper, we investigate learning temporal abstractions in cooperative multi-agent systems, using the options framework (Sutton et al, 1999). First, we address the planning problem for the decentralized POMDP represented by the multi-agent system, by introducing a \emph{common information approach}. We use the notion of \emph{common beliefs} and broadcasting to solve an equivalent centralized POMDP problem. Then, we propose the Distributed Option Critic (DOC) algorithm, which uses centralized option evaluation and decentralized intra-option improvement. We theoretically analyze the asymptotic convergence of DOC and build a new multi-agent environment to demonstrate its validity. Our experiments empirically show that DOC performs competitively against baselines and scales with the number of agents.