Inducing Cooperative behaviour in Sequential-Social dilemmas through Multi-Agent Reinforcement Learning using Status-Quo Loss
This addresses the challenge of inducing cooperation in multi-agent systems for applications like autonomous systems or social simulations, though it appears incremental as it builds on existing methods with specific modifications.
The paper tackled the problem of Deep Reinforcement Learning agents converging to selfish behavior in sequential social dilemmas by introducing a status-quo loss (SQLoss) to encourage cooperative behavior, showing that agents trained with SQLoss evolve cooperation in matrix games and, combined with GameDistill for visual input games like Coin Game, achieve socially desirable outcomes.
In social dilemma situations, individual rationality leads to sub-optimal group outcomes. Several human engagements can be modeled as a sequential (multi-step) social dilemmas. However, in contrast to humans, Deep Reinforcement Learning agents trained to optimize individual rewards in sequential social dilemmas converge to selfish, mutually harmful behavior. We introduce a status-quo loss (SQLoss) that encourages an agent to stick to the status quo, rather than repeatedly changing its policy. We show how agents trained with SQLoss evolve cooperative behavior in several social dilemma matrix games. To work with social dilemma games that have visual input, we propose GameDistill. GameDistill uses self-supervision and clustering to automatically extract cooperative and selfish policies from a social dilemma game. We combine GameDistill and SQLoss to show how agents evolve socially desirable cooperative behavior in the Coin Game.