Cooperative Actor-Critic via TD Error Aggregation
This addresses privacy and communication challenges in decentralized multi-agent systems, offering a scalable solution for large networks, though it is incremental as it builds on existing actor-critic methods.
The paper tackles decentralized cooperative multi-agent reinforcement learning by introducing an actor-critic algorithm with TD error aggregation that respects privacy and handles communication delays and dropouts, achieving convergence to maximize a team-average objective function with a communication burden quadratic in graph size.
In decentralized cooperative multi-agent reinforcement learning, agents can aggregate information from one another to learn policies that maximize a team-average objective function. Despite the willingness to cooperate with others, the individual agents may find direct sharing of information about their local state, reward, and value function undesirable due to privacy issues. In this work, we introduce a decentralized actor-critic algorithm with TD error aggregation that does not violate privacy issues and assumes that communication channels are subject to time delays and packet dropouts. The cost we pay for making such weak assumptions is an increased communication burden for every agent as measured by the dimension of the transmitted data. Interestingly, the communication burden is only quadratic in the graph size, which renders the algorithm applicable in large networks. We provide a convergence analysis under diminishing step size to verify that the agents maximize the team-average objective function.