Environment Complexity and Nash Equilibria in a Sequential Social Dilemma
This work addresses the challenge of achieving optimal cooperation in MARL for general-sum games, which is incremental as it builds on existing matrix game models by adding complexity.
The study tackled the problem of suboptimal outcomes in multi-agent reinforcement learning (MARL) for general-sum games by extending matrix game social dilemmas into more complex, higher-dimensional environments, finding that as complexity increases, agents converge to suboptimal strategies consistent with risk-dominant Nash equilibria.
Multi-agent reinforcement learning (MARL) methods, while effective in zero-sum or positive-sum games, often yield suboptimal outcomes in general-sum games where cooperation is essential for achieving globally optimal outcomes. Matrix game social dilemmas, which abstract key aspects of general-sum interactions, such as cooperation, risk, and trust, fail to model the temporal and spatial dynamics characteristic of real-world scenarios. In response, our study extends matrix game social dilemmas into more complex, higher-dimensional MARL environments. We adapt a gridworld implementation of the Stag Hunt dilemma to more closely match the decision-space of a one-shot matrix game while also introducing variable environment complexity. Our findings indicate that as complexity increases, MARL agents trained in these environments converge to suboptimal strategies, consistent with the risk-dominant Nash equilibria strategies found in matrix games. Our work highlights the impact of environment complexity on achieving optimal outcomes in higher-dimensional game-theoretic MARL environments.