Decentralized Multi-Agent Actor-Critic with Generative Inference
This addresses a specific problem for multi-agent reinforcement learning systems where communication disruptions occur, offering an incremental improvement over existing methods.
The paper tackles the problem of multi-agent systems performing poorly when communications are disrupted during decentralized execution, by augmenting centralized training with generative modeling to infer other agents' observations. The result shows that decentralized training on inferred observations performs as well or better than existing actor-critic methods on three tasks requiring combined local and remote observations.
Recent multi-agent actor-critic methods have utilized centralized training with decentralized execution to address the non-stationarity of co-adapting agents. This training paradigm constrains learning to the centralized phase such that only pre-learned policies may be used during the decentralized phase, which performs poorly when agent communications are delayed, noisy, or disrupted. In this work, we propose a new system that can gracefully handle partially-observable information due to communication disruptions during decentralized execution. Our approach augments the multi-agent actor-critic method's centralized training phase with generative modeling so that agents may infer other agents' observations when provided with locally available context. Our method is evaluated on three tasks that require agents to combine local and remote observations communicated by other agents. We evaluate our approach by introducing both partial observability during decentralized execution, and show that decentralized training on inferred observations performs as well or better than existing actor-critic methods.