Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games
This addresses the problem of fragmented and simplistic evaluation for researchers in multi-agent systems, though it is incremental as it standardizes rather than introduces new methods.
The authors tackled the lack of standardized evaluation in Mean Field Games (MFGs) and Reinforcement Learning (RL) by proposing Bench-MFG, a benchmark suite with problem classes and random instance generation, resulting in guidelines for future comparisons based on empirical testing of algorithms.
The intersection of Mean Field Games (MFGs) and Reinforcement Learning (RL) has fostered a growing family of algorithms designed to solve large-scale multi-agent systems. However, the field currently lacks a standardized evaluation protocol, forcing researchers to rely on bespoke, isolated, and often simplistic environments. This fragmentation makes it difficult to assess the robustness, generalization, and failure modes of emerging methods. To address this gap, we propose a comprehensive benchmark suite for MFGs (Bench-MFG), focusing on the discrete-time, discrete-space, stationary setting for the sake of clarity. We introduce a taxonomy of problem classes, ranging from no-interaction and monotone games to potential and dynamics-coupled games, and provide prototypical environments for each. Furthermore, we propose MF-Garnets, a method for generating random MFG instances to facilitate rigorous statistical testing. We benchmark a variety of learning algorithms across these environments, including a novel black-box approach (MF-PSO) for exploitability minimization. Based on our extensive empirical results, we propose guidelines to standardize future experimental comparisons. Code available at \href{https://github.com/lorenzomagnino/Bench-MFG}{https://github.com/lorenzomagnino/Bench-MFG}.