MAVEN: Multi-Agent Variational Exploration
This addresses exploration issues in cooperative multi-agent systems, which is crucial for applications like robotics and game AI, though it is an incremental improvement over existing methods.
The paper tackles the problem of poor exploration and suboptimality in value-based multi-agent reinforcement learning methods like QMIX, and proposes MAVEN, a hybrid approach that introduces a latent space for hierarchical control, resulting in significant performance improvements on the challenging SMAC domain.
Centralised training with decentralised execution is an important setting for cooperative deep multi-agent reinforcement learning due to communication constraints during execution and computational tractability in training. In this paper, we analyse value-based methods that are known to have superior performance in complex environments [43]. We specifically focus on QMIX [40], the current state-of-the-art in this domain. We show that the representational constraints on the joint action-values introduced by QMIX and similar methods lead to provably poor exploration and suboptimality. Furthermore, we propose a novel approach called MAVEN that hybridises value and policy-based methods by introducing a latent space for hierarchical control. The value-based agents condition their behaviour on the shared latent variable controlled by a hierarchical policy. This allows MAVEN to achieve committed, temporally extended exploration, which is key to solving complex multi-agent tasks. Our experimental results show that MAVEN achieves significant performance improvements on the challenging SMAC domain [43].