The Advantage Regret-Matching Actor-Critic
This work addresses the challenge of efficient and stable no-regret learning in reinforcement learning, particularly for large-scale and multi-agent settings, though it appears incremental as it builds on existing regret minimization techniques.
The paper tackles the problem of model-free reinforcement learning by introducing a no-regret method that replays past policies to reconstruct hindsight value estimates, which are used to predict conditional advantages and update policies via regret matching. It shows that ARMAC avoids excessive variance in large environments and approaches Nash equilibria in multi-agent benchmarks, with exploitability estimates provided for a large poker game.
Regret minimization has played a key role in online learning, equilibrium computation in games, and reinforcement learning (RL). In this paper, we describe a general model-free RL method for no-regret learning based on repeated reconsideration of past behavior. We propose a model-free RL algorithm, the AdvantageRegret-Matching Actor-Critic (ARMAC): rather than saving past state-action data, ARMAC saves a buffer of past policies, replaying through them to reconstruct hindsight assessments of past behavior. These retrospective value estimates are used to predict conditional advantages which, combined with regret matching, produces a new policy. In particular, ARMAC learns from sampled trajectories in a centralized training setting, without requiring the application of importance sampling commonly used in Monte Carlo counterfactual regret (CFR) minimization; hence, it does not suffer from excessive variance in large environments. In the single-agent setting, ARMAC shows an interesting form of exploration by keeping past policies intact. In the multiagent setting, ARMAC in self-play approaches Nash equilibria on some partially-observable zero-sum benchmarks. We provide exploitability estimates in the significantly larger game of betting-abstracted no-limit Texas Hold'em.