Mean-Field Reinforcement Learning without Synchrony
This addresses the limitation of requiring synchrony in MF-RL for large populations, enabling applications in asynchronous multi-agent systems, though it is an incremental extension of existing MF-RL theory.
The paper tackles the problem of asynchrony in mean-field reinforcement learning (MF-RL) by introducing the Temporal Mean Field (TMF) framework, which uses the population distribution instead of the mean action to handle scenarios where agents act at different times, and proves theoretical guarantees and demonstrates in experiments that TMF-PG achieves near-identical performance with error decaying at an O(1/√N) rate.
Mean-field reinforcement learning (MF-RL) scales multi-agent RL to large populations by reducing each agent's dependence on others to a single summary statistic -- the mean action. However, this reduction requires every agent to act at every time step; when some agents are idle, the mean action is simply undefined. Addressing asynchrony therefore requires a different summary statistic -- one that remains defined regardless of which agents act. The population distribution $μ\in Δ(\mathcal{O})$ -- the fraction of agents at each observation -- satisfies this requirement: its dimension is independent of $N$, and under exchangeability it fully determines each agent's reward and transition. Existing MF-RL theory, however, is built on the mean action and does not extend to $μ$. We therefore construct the Temporal Mean Field (TMF) framework around the population distribution $μ$ from scratch, covering the full spectrum from fully synchronous to purely sequential decision-making within a single theory. We prove existence and uniqueness of TMF equilibria, establish an $O(1/\sqrt{N})$ finite-population approximation bound that holds regardless of how many agents act per step, and prove convergence of a policy gradient algorithm (TMF-PG) to the unique equilibrium. Experiments on a resource selection game and a dynamic queueing game confirm that TMF-PG achieves near-identical performance whether one agent or all $N$ act per step, with approximation error decaying at the predicted $O(1/\sqrt{N})$ rate.