Approximated Multi-Agent Fitted Q Iteration
This addresses a scalability bottleneck for researchers and practitioners in multi-agent systems, offering a more efficient alternative to existing methods.
The paper tackles the computational intractability of multi-agent batch reinforcement learning by proposing AMAFQI, which reduces computation time from exponential to linear scaling with the number of agents while maintaining similar performance to FQI in simulations.
We formulate an efficient approximation for multi-agent batch reinforcement learning, the approximated multi-agent fitted Q iteration (AMAFQI). We present a detailed derivation of our approach. We propose an iterative policy search and show that it yields a greedy policy with respect to multiple approximations of the centralized, learned Q-function. In each iteration and policy evaluation, AMAFQI requires a number of computations that scales linearly with the number of agents whereas the analogous number of computations increase exponentially for the fitted Q iteration (FQI), a commonly used approaches in batch reinforcement learning. This property of AMAFQI is fundamental for the design of a tractable multi-agent approach. We evaluate the performance of AMAFQI and compare it to FQI in numerical simulations. The simulations illustrate the significant computation time reduction when using AMAFQI instead of FQI in multi-agent problems and corroborate the similar performance of both approaches.