Homogenization of Multi-agent Learning Dynamics in Finite-state Markov Games
This provides a tractable method for analyzing multi-agent reinforcement learning dynamics, but it is incremental as it builds on existing homogenization techniques.
The paper tackles the problem of approximating learning dynamics for multiple reinforcement learning agents in finite-state Markov games by rescaling the learning process, proving convergence to an ordinary differential equation (ODE) that provides a deterministic approximation.
This paper introduces a new approach for approximating the learning dynamics of multiple reinforcement learning (RL) agents interacting in a finite-state Markov game. The idea is to rescale the learning process by simultaneously reducing the learning rate and increasing the update frequency, effectively treating the agent's parameters as a slow-evolving variable influenced by the fast-mixing game state. Under mild assumptions-ergodicity of the state process and continuity of the updates-we prove the convergence of this rescaled process to an ordinary differential equation (ODE). This ODE provides a tractable, deterministic approximation of the agent's learning dynamics. An implementation of the framework is available at\,: https://github.com/yannKerzreho/MarkovGameApproximation