Policy Gradient With Value Function Approximation For Collective Multiagent Planning
This work addresses the challenge of optimizing policies for multiagent systems where collective behavior affects rewards and dynamics, offering incremental improvements in computational tractability.
The paper tackles the problem of collective multiagent planning in a subclass of decentralized POMDPs by proposing an actor-critic reinforcement learning method, which achieves better quality solutions than previous approaches on synthetic and real-world benchmarks.
Decentralized (PO)MDPs provide an expressive framework for sequential decision making in a multiagent system. Given their computational complexity, recent research has focused on tractable yet practical subclasses of Dec-POMDPs. We address such a subclass called CDEC-POMDP where the collective behavior of a population of agents affects the joint-reward and environment dynamics. Our main contribution is an actor-critic (AC) reinforcement learning method for optimizing CDEC-POMDP policies. Vanilla AC has slow convergence for larger problems. To address this, we show how a particular decomposition of the approximate action-value function over agents leads to effective updates, and also derive a new way to train the critic based on local reward signals. Comparisons on a synthetic benchmark and a real-world taxi fleet optimization problem show that our new AC approach provides better quality solutions than previous best approaches.