The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces
This work addresses a key gap in reinforcement learning theory by providing a provable framework for multi-agent scenarios, which is incremental as it adapts existing single-agent concepts to the multi-agent domain.
The paper tackles the challenge of extending provable reinforcement learning to multi-agent settings with large state spaces by proposing a new algorithm that provably finds Nash equilibrium policies in two-player zero-sum Markov Games using a polynomial number of samples, based on a new complexity measure called the multi-agent Bellman-Eluder dimension.
Modern reinforcement learning (RL) commonly engages practical problems with large state spaces, where function approximation must be deployed to approximate either the value function or the policy. While recent progresses in RL theory address a rich set of RL problems with general function approximation, such successes are mostly restricted to the single-agent setting. It remains elusive how to extend these results to multi-agent RL, especially due to the new challenges arising from its game-theoretical nature. This paper considers two-player zero-sum Markov Games (MGs). We propose a new algorithm that can provably find the Nash equilibrium policy using a polynomial number of samples, for any MG with low multi-agent Bellman-Eluder dimension -- a new complexity measure adapted from its single-agent version (Jin et al., 2021). A key component of our new algorithm is the exploiter, which facilitates the learning of the main player by deliberately exploiting her weakness. Our theoretical framework is generic, which applies to a wide range of models including but not limited to tabular MGs, MGs with linear or kernel function approximation, and MGs with rich observations.