Multi-Agent MDP Homomorphic Networks
This work addresses data efficiency in cooperative multi-agent reinforcement learning, offering a novel factorization method for distributed execution, though it is incremental in extending symmetry concepts from single-agent to multi-agent settings.
The paper tackles the problem of leveraging global symmetries in cooperative multi-agent systems for improved data efficiency, proposing Multi-Agent MDP Homomorphic Networks that enable distributed execution with local information, and shows empirically that these policies enhance data efficiency compared to non-equivariant baselines.
This paper introduces Multi-Agent MDP Homomorphic Networks, a class of networks that allows distributed execution using only local information, yet is able to share experience between global symmetries in the joint state-action space of cooperative multi-agent systems. In cooperative multi-agent systems, complex symmetries arise between different configurations of the agents and their local observations. For example, consider a group of agents navigating: rotating the state globally results in a permutation of the optimal joint policy. Existing work on symmetries in single agent reinforcement learning can only be generalized to the fully centralized setting, because such approaches rely on the global symmetry in the full state-action spaces, and these can result in correspondences across agents. To encode such symmetries while still allowing distributed execution we propose a factorization that decomposes global symmetries into local transformations. Our proposed factorization allows for distributing the computation that enforces global symmetries over local agents and local interactions. We introduce a multi-agent equivariant policy network based on this factorization. We show empirically on symmetric multi-agent problems that globally symmetric distributable policies improve data efficiency compared to non-equivariant baselines.