Revisiting the Master-Slave Architecture in Multi-Agent Deep Reinforcement Learning
This work addresses the problem of improving collaboration efficiency in multi-agent systems for AI researchers and practitioners, though it is incremental as it builds on existing hierarchical ideas.
The paper tackled the challenge of integrating decentralized and centralized perspectives in multi-agent deep reinforcement learning by revisiting the master-slave architecture, resulting in consistent outperformance of latest methods in synthetic experiments and StarCraft micromanagement tasks.
Many tasks in artificial intelligence require the collaboration of multiple agents. We exam deep reinforcement learning for multi-agent domains. Recent research efforts often take the form of two seemingly conflicting perspectives, the decentralized perspective, where each agent is supposed to have its own controller; and the centralized perspective, where one assumes there is a larger model controlling all agents. In this regard, we revisit the idea of the master-slave architecture by incorporating both perspectives within one framework. Such a hierarchical structure naturally leverages advantages from one another. The idea of combining both perspectives is intuitive and can be well motivated from many real world systems, however, out of a variety of possible realizations, we highlights three key ingredients, i.e. composed action representation, learnable communication and independent reasoning. With network designs to facilitate these explicitly, our proposal consistently outperforms latest competing methods both in synthetic experiments and when applied to challenging StarCraft micromanagement tasks.