QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
This addresses performance bottlenecks in cooperative multi-agent reinforcement learning for applications like gaming or robotics, but it is incremental as it builds directly on QTRAN.
The paper tackled the poor empirical performance of QTRAN in complex multi-agent environments like Starcraft Multi-Agent Challenge (SMAC) by proposing QTRAN++, which stabilizes training, removes role separation, and adds a multi-head mixing network, resulting in state-of-the-art performance in SMAC.
QTRAN is a multi-agent reinforcement learning (MARL) algorithm capable of learning the largest class of joint-action value functions up to date. However, despite its strong theoretical guarantee, it has shown poor empirical performance in complex environments, such as Starcraft Multi-Agent Challenge (SMAC). In this paper, we identify the performance bottleneck of QTRAN and propose a substantially improved version, coined QTRAN++. Our gains come from (i) stabilizing the training objective of QTRAN, (ii) removing the strict role separation between the action-value estimators of QTRAN, and (iii) introducing a multi-head mixing network for value transformation. Through extensive evaluation, we confirm that our diagnosis is correct, and QTRAN++ successfully bridges the gap between empirical performance and theoretical guarantee. In particular, QTRAN++ newly achieves state-of-the-art performance in the SMAC environment. The code will be released.