Finite-Sample Analysis of Decentralized Q-Learning for Stochastic Games
This work addresses the challenge of sample-efficient learning in decentralized multi-agent systems where agents cannot observe others, which is incremental as it extends existing Q-learning analyses to a broader class of games.
The paper tackles the problem of decentralized multi-agent reinforcement learning in stochastic games by establishing finite-sample complexity bounds for fully decentralized Q-learning algorithms in weakly acyclic general-sum games, including cooperative settings, with results showing convergence to equilibria in both tabular and linear function approximation cases.
Learning in stochastic games is arguably the most standard and fundamental setting in multi-agent reinforcement learning (MARL). In this paper, we consider decentralized MARL in stochastic games in the non-asymptotic regime. In particular, we establish the finite-sample complexity of fully decentralized Q-learning algorithms in a significant class of general-sum stochastic games (SGs) - weakly acyclic SGs, which includes the common cooperative MARL setting with an identical reward to all agents (a Markov team problem) as a special case. We focus on the practical while challenging setting of fully decentralized MARL, where neither the rewards nor the actions of other agents can be observed by each agent. In fact, each agent is completely oblivious to the presence of other decision makers. Both the tabular and the linear function approximation cases have been considered. In the tabular setting, we analyze the sample complexity for the decentralized Q-learning algorithm to converge to a Markov perfect equilibrium (Nash equilibrium). With linear function approximation, the results are for convergence to a linear approximated equilibrium - a new notion of equilibrium that we propose - which describes that each agent's policy is a best reply (to other agents) within a linear space. Numerical experiments are also provided for both settings to demonstrate the results.