Best Response Shaping
This work expands the applicability of multi-agent RL in partially competitive environments, providing a new pathway towards improved social welfare in general sum games, though it appears incremental as it builds on prior methods like LOLA and POLA.
The paper tackles the challenge of fostering reciprocity-based cooperation in multi-agent deep reinforcement learning in partially competitive environments, where existing methods like LOLA and POLA are exploitable by opponents that optimize over many steps. The authors introduce Best Response Shaping (BRS), which differentiates through an opponent approximating the best response, and demonstrate enhanced performance against a Monte Carlo Tree Search opponent in the Coin Game.
We investigate the challenge of multi-agent deep reinforcement learning in partially competitive environments, where traditional methods struggle to foster reciprocity-based cooperation. LOLA and POLA agents learn reciprocity-based cooperative policies by differentiation through a few look-ahead optimization steps of their opponent. However, there is a key limitation in these techniques. Because they consider a few optimization steps, a learning opponent that takes many steps to optimize its return may exploit them. In response, we introduce a novel approach, Best Response Shaping (BRS), which differentiates through an opponent approximating the best response, termed the "detective." To condition the detective on the agent's policy for complex games we propose a state-aware differentiable conditioning mechanism, facilitated by a question answering (QA) method that extracts a representation of the agent based on its behaviour on specific environment states. To empirically validate our method, we showcase its enhanced performance against a Monte Carlo Tree Search (MCTS) opponent, which serves as an approximation to the best response in the Coin Game. This work expands the applicability of multi-agent RL in partially competitive environments and provides a new pathway towards achieving improved social welfare in general sum games.