Provably Convergent Actor-Critic in Risk-averse MARL
This addresses a fundamental open problem in Multi-Agent Reinforcement Learning for scenarios requiring practical stationary policies, with incremental improvements in convergence and risk handling.
The paper tackles the problem of learning stationary policies in infinite-horizon general-sum Markov games, which is computationally intractable for classic equilibria, by introducing Risk-averse Quantal response Equilibria (RQE) and a two-timescale Actor-Critic algorithm, achieving global convergence with finite-sample guarantees and empirically showing superior convergence compared to risk-neutral baselines.
Learning stationary policies in infinite-horizon general-sum Markov games (MGs) remains a fundamental open problem in Multi-Agent Reinforcement Learning (MARL). While stationary strategies are preferred for their practicality, computing stationary forms of classic game-theoretic equilibria is computationally intractable -- a stark contrast to the comparative ease of solving single-agent RL or zero-sum games. To bridge this gap, we study Risk-averse Quantal response Equilibria (RQE), a solution concept rooted in behavioral game theory that incorporates risk aversion and bounded rationality. We demonstrate that RQE possesses strong regularity conditions that make it uniquely amenable to learning in MGs. We propose a novel two-timescale Actor-Critic algorithm characterized by a fast-timescale actor and a slow-timescale critic. Leveraging the regularity of RQE, we prove that this approach achieves global convergence with finite-sample guarantees. We empirically validate our algorithm in several environments to demonstrate superior convergence properties compared to risk-neutral baselines.