Efficiently Quantifying Individual Agent Importance in Cooperative MARL
This provides an efficient explainability tool for diagnosing failures in MARL systems, though it is incremental as it builds on existing difference rewards and Shapley value concepts.
The paper tackles the challenge of efficiently quantifying individual agent contributions in cooperative multi-agent reinforcement learning (MARL) by adapting difference rewards into a method called Agent Importance, which reduces computational complexity from exponential to linear relative to the number of agents and shows strong correlation with true Shapley values and individual rewards.
Measuring the contribution of individual agents is challenging in cooperative multi-agent reinforcement learning (MARL). In cooperative MARL, team performance is typically inferred from a single shared global reward. Arguably, among the best current approaches to effectively measure individual agent contributions is to use Shapley values. However, calculating these values is expensive as the computational complexity grows exponentially with respect to the number of agents. In this paper, we adapt difference rewards into an efficient method for quantifying the contribution of individual agents, referred to as Agent Importance, offering a linear computational complexity relative to the number of agents. We show empirically that the computed values are strongly correlated with the true Shapley values, as well as the true underlying individual agent rewards, used as the ground truth in environments where these are available. We demonstrate how Agent Importance can be used to help study MARL systems by diagnosing algorithmic failures discovered in prior MARL benchmarking work. Our analysis illustrates Agent Importance as a valuable explainability component for future MARL benchmarks.