SMIX($λ$): Enhancing Centralized Value Functions for Cooperative Multi-Agent Reinforcement Learning
This work addresses the curse of dimensionality in multi-agent reinforcement learning for cooperative tasks, offering an incremental improvement over existing methods.
The paper tackles the challenge of learning stable and generalizable centralized value functions in multi-agent reinforcement learning by proposing SMIX(λ), which uses λ-returns to compute TD errors and a modified QMIX network. Experiments on the StarCraft Multi-Agent Challenge show that SMIX(λ) outperforms state-of-the-art methods by a large margin and can enhance other CTDE-type algorithms.
Learning a stable and generalizable centralized value function (CVF) is a crucial but challenging task in multi-agent reinforcement learning (MARL), as it has to deal with the issue that the joint action space increases exponentially with the number of agents in such scenarios. This paper proposes an approach, named SMIX($λ$), to address the issue using an efficient off-policy centralized training method within a flexible learner search space. As importance sampling for such off-policy training is both computationally costly and numerically unstable, we proposed to use the $λ$-return as a proxy to compute the TD error. With this new loss function objective, we adopt a modified QMIX network structure as the base to train our model. By further connecting it with the ${Q(λ)}$ approach from an unified expectation correction viewpoint, we show that the proposed SMIX($λ$) is equivalent to ${Q(λ)}$ and hence shares its convergence properties, while without being suffered from the aforementioned curse of dimensionality problem inherent in MARL. Experiments on the StarCraft Multi-Agent Challenge (SMAC) benchmark demonstrate that our approach not only outperforms several state-of-the-art MARL methods by a large margin, but also can be used as a general tool to improve the overall performance of other CTDE-type algorithms by enhancing their CVFs.