On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximation
This work addresses the fundamental statistical efficiency problem for researchers in mean-field reinforcement learning, offering a novel complexity measure and algorithms with improved sample complexity under minimal assumptions.
The paper tackles the statistical efficiency of Reinforcement Learning in Mean-Field Control and Mean-Field Game settings with general function approximation, introducing the Mean-Field Model-Based Eluder Dimension (MF-MBED) to characterize complexity and showing that many problems have low MF-MBED, with algorithms achieving ε-optimal policies with sample complexity polynomial in MF-MBED, potentially much lower than state-action space size.
In this paper, we study the fundamental statistical efficiency of Reinforcement Learning in Mean-Field Control (MFC) and Mean-Field Game (MFG) with general model-based function approximation. We introduce a new concept called Mean-Field Model-Based Eluder Dimension (MF-MBED), which characterizes the inherent complexity of mean-field model classes. We show that a rich family of Mean-Field RL problems exhibits low MF-MBED. Additionally, we propose algorithms based on maximal likelihood estimation, which can return an $ε$-optimal policy for MFC or an $ε$-Nash Equilibrium policy for MFG. The overall sample complexity depends only polynomially on MF-MBED, which is potentially much lower than the size of state-action space. Compared with previous works, our results only require the minimal assumptions including realizability and Lipschitz continuity.