Quantifying Agent Interaction in Multi-agent Reinforcement Learning for Cost-efficient Generalization
This work addresses generalization issues in MARL for researchers and practitioners, but it is incremental as it builds on existing methods with a new metric and allocation strategy.
The paper tackles the challenge of generalization in Multi-agent Reinforcement Learning by introducing the Level of Influence (LoI) metric to quantify agent interactions, showing that strategic resource allocation based on LoI achieves higher performance than uniform allocation under the same computation budget.
Generalization poses a significant challenge in Multi-agent Reinforcement Learning (MARL). The extent to which an agent is influenced by unseen co-players depends on the agent's policy and the specific scenario. A quantitative examination of this relationship sheds light on effectively training agents for diverse scenarios. In this study, we present the Level of Influence (LoI), a metric quantifying the interaction intensity among agents within a given scenario and environment. We observe that, generally, a more diverse set of co-play agents during training enhances the generalization performance of the ego agent; however, this improvement varies across distinct scenarios and environments. LoI proves effective in predicting these improvement disparities within specific scenarios. Furthermore, we introduce a LoI-guided resource allocation method tailored to train a set of policies for diverse scenarios under a constrained budget. Our results demonstrate that strategic resource allocation based on LoI can achieve higher performance than uniform allocation under the same computation budget.