NePPO: Near-Potential Policy Optimization for General-Sum Multi-Agent Reinforcement Learning
This work provides a new approach for achieving more stable learning and approximate Nash equilibria in general-sum MARL, which is a significant problem for researchers and practitioners developing multi-agent systems.
This paper addresses the challenge of unstable learning dynamics and lack of convergence guarantees in general-sum multi-agent reinforcement learning (MARL). The authors propose Near-Potential Policy Optimization (NePPO), which learns a player-independent potential function to approximate Nash equilibria in mixed cooperative-competitive environments, demonstrating superior performance compared to baselines like MAPPO, IPPO, and MADDPG.
Multi-agent reinforcement learning (MARL) is increasingly used to design learning-enabled agents that interact in shared environments. However, training MARL algorithms in general-sum games remains challenging: learning dynamics can become unstable, and convergence guarantees typically hold only in restricted settings such as two-player zero-sum or fully cooperative games. Moreover, when agents have heterogeneous and potentially conflicting preferences, it is unclear what system-level objective should guide learning. In this paper, we propose a new MARL pipeline called Near-Potential Policy Optimization (NePPO) for computing approximate Nash equilibria in mixed cooperative--competitive environments. The core idea is to learn a player-independent potential function such that the Nash equilibrium of a cooperative game with this potential as the common utility approximates a Nash equilibrium of the original game. To this end, we introduce a novel MARL objective such that minimizing this objective yields the best possible potential function candidate and consequently an approximate Nash equilibrium of the original game. We develop an algorithmic pipeline that minimizes this objective using zeroth-order gradient descent and returns an approximate Nash equilibrium policy. We empirically show the superior performance of this approach compared to popular baselines such as MAPPO, IPPO and MADDPG.