Latent Interactive A2C for Improved RL in Open Many-Agent Systems
This work addresses the problem of decentralized training in open many-agent systems for researchers in MARL, but it is incremental as it builds on existing IA2C methods.
The paper tackles the challenge of multiagent reinforcement learning in competitive or adversarial settings by proposing latent IA2C, which uses an encoder-decoder to learn latent representations, resulting in significant improvements in sample efficiency, such as reduced variance and faster convergence.
There is a prevalence of multiagent reinforcement learning (MARL) methods that engage in centralized training. But, these methods involve obtaining various types of information from the other agents, which may not be feasible in competitive or adversarial settings. A recent method, the interactive advantage actor critic (IA2C), engages in decentralized training coupled with decentralized execution, aiming to predict the other agents' actions from possibly noisy observations. In this paper, we present the latent IA2C that utilizes an encoder-decoder architecture to learn a latent representation of the hidden state and other agents' actions. Our experiments in two domains -- each populated by many agents -- reveal that the latent IA2C significantly improves sample efficiency by reducing variance and converging faster. Additionally, we introduce open versions of these domains where the agent population may change over time, and evaluate on these instances as well.