MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning
This work addresses the challenge of quantifying long-term causal influence for coordination in multi-agent reinforcement learning, offering a significant performance boost over existing methods.
MAGIC introduces a framework that extracts multi-step causal influences between agents and converts them into intrinsic rewards to promote coordination in MARL, achieving at least 10.1% improvement over state-of-the-art methods on MPE and SMAC benchmarks.
A key challenge in multi-agent reinforcement learning (MARL) lies in designing learning signals that effectively promote coordination among agents. Designing such signals necessitates the ability to quantify the true, long-term causal influence between agents. To address this, we introduce Multi-step Advantage-Gated Interventional Causal MARL (MAGIC), a framework that extracts multi-step causal influences between agents and selectively converts them into intrinsic rewards. MAGIC uses causal intervention with conditional mutual information to quantify long-horizon agent influence, and introduces an advantage-based gating mechanism to ensure exploration is directed toward beneficial, goal-aligned behaviors. Experiments across multiple standard MARL benchmarks and task families, including MPE and SMAC/SMACv2, demonstrate that MAGIC outperforms state-of-the-art methods by a significant margin, achieving an improvement of at least 10.1% in the main evaluation metric.