Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning
This work addresses credit assignment in cooperative multi-agent systems for applications like gaming and robotics, representing an incremental improvement by hybridizing existing approaches.
The paper tackles cooperative multi-agent reinforcement learning by proposing a decomposed multi-agent soft actor-critic (mSAC) method that combines value function decomposition and policy gradients, achieving competitive results with state-of-the-art methods like Qmix on StarCraft II tasks and performing well on large action space scenarios.
Deep reinforcement learning methods have shown great performance on many challenging cooperative multi-agent tasks. Two main promising research directions are multi-agent value function decomposition and multi-agent policy gradients. In this paper, we propose a new decomposed multi-agent soft actor-critic (mSAC) method, which effectively combines the advantages of the aforementioned two methods. The main modules include decomposed Q network architecture, discrete probabilistic policy and counterfactual advantage function (optinal). Theoretically, mSAC supports efficient off-policy learning and addresses credit assignment problem partially in both discrete and continuous action spaces. Tested on StarCraft II micromanagement cooperative multiagent benchmark, we empirically investigate the performance of mSAC against its variants and analyze the effects of the different components. Experimental results demonstrate that mSAC significantly outperforms policy-based approach COMA, and achieves competitive results with SOTA value-based approach Qmix on most tasks in terms of asymptotic perfomance metric. In addition, mSAC achieves pretty good results on large action space tasks, such as 2c_vs_64zg and MMM2.