Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning
This work addresses the problem of scalable reinforcement learning for large populations of agents, providing a theoretical foundation for efficient learning in multi-agent systems, though it is incremental in extending model-based methods to the mean-field setting.
The paper tackles the challenge of learning in multi-agent systems with unknown dynamics by proposing an efficient model-based reinforcement learning algorithm, $M^3-UCRL$, which provably solves the Mean-Field Control problem and achieves the first general regret bounds for this setting.
Learning in multi-agent systems is highly challenging due to several factors including the non-stationarity introduced by agents' interactions and the combinatorial nature of their state and action spaces. In particular, we consider the Mean-Field Control (MFC) problem which assumes an asymptotically infinite population of identical agents that aim to collaboratively maximize the collective reward. In many cases, solutions of an MFC problem are good approximations for large systems, hence, efficient learning for MFC is valuable for the analogous discrete agent setting with many agents. Specifically, we focus on the case of unknown system dynamics where the goal is to simultaneously optimize for the rewards and learn from experience. We propose an efficient model-based reinforcement learning algorithm, $M^3-UCRL$, that runs in episodes, balances between exploration and exploitation during policy learning, and provably solves this problem. Our main theoretical contributions are the first general regret bounds for model-based reinforcement learning for MFC, obtained via a novel mean-field type analysis. To learn the system's dynamics, $M^3-UCRL$ can be instantiated with various statistical models, e.g., neural networks or Gaussian Processes. Moreover, we provide a practical parametrization of the core optimization problem that facilitates gradient-based optimization techniques when combined with differentiable dynamics approximation methods such as neural networks.