Multi-Agent Determinantal Q-Learning
This addresses a key limitation in multi-agent learning for cooperative tasks, though it appears incremental as it builds on existing paradigms.
The paper tackles the problem of restrictive assumptions in centralized training with decentralized execution for multi-agent systems by proposing multi-agent determinantal Q-learning, which eliminates the need for a priori structural constraints and generalizes existing methods like VDN, QMIX, and QTRAN, demonstrating effectiveness on cooperative benchmarks.
Centralized training with decentralized execution has become an important paradigm in multi-agent learning. Though practical, current methods rely on restrictive assumptions to decompose the centralized value function across agents for execution. In this paper, we eliminate this restriction by proposing multi-agent determinantal Q-learning. Our method is established on Q-DPP, an extension of determinantal point process (DPP) with partition-matroid constraint to multi-agent setting. Q-DPP promotes agents to acquire diverse behavioral models; this allows a natural factorization of the joint Q-functions with no need for \emph{a priori} structural constraints on the value function or special network architectures. We demonstrate that Q-DPP generalizes major solutions including VDN, QMIX, and QTRAN on decentralizable cooperative tasks. To efficiently draw samples from Q-DPP, we adopt an existing sample-by-projection sampler with theoretical approximation guarantee. The sampler also benefits exploration by coordinating agents to cover orthogonal directions in the state space during multi-agent training. We evaluate our algorithm on various cooperative benchmarks; its effectiveness has been demonstrated when compared with the state-of-the-art.