Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning
This addresses adaptability issues in cooperative multi-agent reinforcement learning for researchers and practitioners, though it is incremental as it builds on existing value decomposition approaches.
The paper tackles the problem of value decomposition methods in cooperative multi-agent reinforcement learning struggling to adapt to shifting optima during training, and proposes S2Q, which retains alternative high-value actions to improve adaptability, resulting in consistent outperformance over various MARL algorithms on benchmarks.
Value decomposition is a core approach for cooperative multi-agent reinforcement learning (MARL). However, existing methods still rely on a single optimal action and struggle to adapt when the underlying value function shifts during training, often converging to suboptimal policies. To address this limitation, we propose Successive Sub-value Q-learning (S2Q), which learns multiple sub-value functions to retain alternative high-value actions. Incorporating these sub-value functions into a Softmax-based behavior policy, S2Q encourages persistent exploration and enables $Q^{\text{tot}}$ to adjust quickly to the changing optima. Experiments on challenging MARL benchmarks confirm that S2Q consistently outperforms various MARL algorithms, demonstrating improved adaptability and overall performance. Our code is available at https://github.com/hyeon1996/S2Q.