Social diversity and social preferences in mixed-motive reinforcement learning
This addresses the problem of improving agent performance in mixed-motive scenarios for AI and multi-agent systems, representing an incremental advance by applying known social psychology concepts to reinforcement learning.
The study tackled the effect of population heterogeneity, specifically through Social Value Orientation (SVO), on reinforcement learning in mixed-motive games, demonstrating that heterogeneous populations lead to more generalized and high-performing policies compared to homogeneous ones.
Recent research on reinforcement learning in pure-conflict and pure-common interest games has emphasized the importance of population heterogeneity. In contrast, studies of reinforcement learning in mixed-motive games have primarily leveraged homogeneous approaches. Given the defining characteristic of mixed-motive games--the imperfect correlation of incentives between group members--we study the effect of population heterogeneity on mixed-motive reinforcement learning. We draw on interdependence theory from social psychology and imbue reinforcement learning agents with Social Value Orientation (SVO), a flexible formalization of preferences over group outcome distributions. We subsequently explore the effects of diversity in SVO on populations of reinforcement learning agents in two mixed-motive Markov games. We demonstrate that heterogeneity in SVO generates meaningful and complex behavioral variation among agents similar to that suggested by interdependence theory. Empirical results in these mixed-motive dilemmas suggest agents trained in heterogeneous populations develop particularly generalized, high-performing policies relative to those trained in homogeneous populations.