Greedy UnMixing for Q-Learning in Multi-Agent Reinforcement Learning
This addresses value overestimation issues in cooperative MARL, which is an incremental improvement over existing methods.
The paper tackles the problem of value overestimation in cooperative multi-agent reinforcement learning (MARL) by introducing Greedy UnMix (GUM), a conservative Q-learning method that restricts state-marginal in datasets to avoid unobserved joint state-action spaces. It demonstrates superior performance to existing Q-learning MARL approaches and more general MARL algorithms on benchmark tasks.
This paper introduces Greedy UnMix (GUM) for cooperative multi-agent reinforcement learning (MARL). Greedy UnMix aims to avoid scenarios where MARL methods fail due to overestimation of values as part of the large joint state-action space. It aims to address this through a conservative Q-learning approach through restricting the state-marginal in the dataset to avoid unobserved joint state action spaces, whilst concurrently attempting to unmix or simplify the problem space under the centralized training with decentralized execution paradigm. We demonstrate the adherence to Q-function lower bounds in the Q-learning for MARL scenarios, and demonstrate superior performance to existing Q-learning MARL approaches as well as more general MARL algorithms over a set of benchmark MARL tasks, despite its relative simplicity compared with state-of-the-art approaches.