Understanding Action Effects through Instrumental Empowerment in Multi-Agent Reinforcement Learning
This work addresses the challenge of explainability in MARL systems for researchers and practitioners, though it is incremental as it builds on existing concepts like Shapley values and empowerment.
The paper tackled the problem of understanding individual agent behaviors in multi-agent reinforcement learning without explicit reward signals by introducing Intended Cooperation Values (ICVs), a method based on information-theoretic Shapley values to quantify causal influence on teammates' instrumental empowerment, and demonstrated its ability to identify beneficial behaviors and cooperation dynamics in various tasks.
To reliably deploy Multi-Agent Reinforcement Learning (MARL) systems, it is crucial to understand individual agent behaviors. While prior work typically evaluates overall team performance based on explicit reward signals, it is unclear how to infer agent contributions in the absence of any value feedback. In this work, we investigate whether meaningful insights into agent behaviors can be extracted solely by analyzing the policy distribution. Inspired by the phenomenon that intelligent agents tend to pursue convergent instrumental values, we introduce Intended Cooperation Values (ICVs), a method based on information-theoretic Shapley values for quantifying each agent's causal influence on their co-players' instrumental empowerment. Specifically, ICVs measure an agent's action effect on its teammates' policies by assessing their decision (un)certainty and preference alignment. By analyzing action effects on policies and value functions across cooperative and competitive MARL tasks, our method identifies which agent behaviors are beneficial to team success, either by fostering deterministic decisions or by preserving flexibility for future action choices, while also revealing the extent to which agents adopt similar or diverse strategies. Our proposed method offers novel insights into cooperation dynamics and enhances explainability in MARL systems.