Agent-Centric Representations for Multi-Agent Reinforcement Learning
This work addresses the challenge of relational reasoning in fully cooperative MARL, offering incremental improvements in sample efficiency and generalization for multi-agent systems.
The paper tackled the problem of improving multi-agent reinforcement learning (MARL) by incorporating agent-centric representations, similar to object-centric approaches used in relational reasoning. The result showed that these representations led to more complex cooperation strategies, enhanced sample efficiency, and better generalization in environments like Google Research Football and DeepMind Lab 2D.
Object-centric representations have recently enabled significant progress in tackling relational reasoning tasks. By building a strong object-centric inductive bias into neural architectures, recent efforts have improved generalization and data efficiency of machine learning algorithms for these problems. One problem class involving relational reasoning that still remains under-explored is multi-agent reinforcement learning (MARL). Here we investigate whether object-centric representations are also beneficial in the fully cooperative MARL setting. Specifically, we study two ways of incorporating an agent-centric inductive bias into our RL algorithm: 1. Introducing an agent-centric attention module with explicit connections across agents 2. Adding an agent-centric unsupervised predictive objective (i.e. not using action labels), to be used as an auxiliary loss for MARL, or as the basis of a pre-training step. We evaluate these approaches on the Google Research Football environment as well as DeepMind Lab 2D. Empirically, agent-centric representation learning leads to the emergence of more complex cooperation strategies between agents as well as enhanced sample efficiency and generalization.