Learning When to Cooperate Under Heterogeneous Goals
This work is important for developing more flexible and adaptable AI agents capable of effective collaboration in complex, open-ended environments where goals may not always align.
This paper addresses the problem of agents learning when to cooperate or act alone when they have heterogeneous goals, extending the Ad Hoc Teamwork (AHT) setting. The authors introduce a hierarchical learning approach combining imitation and reinforcement learning, which outperforms baseline methods in two cooperative environments.
A significant element of human cooperative intelligence lies in our ability to identify opportunities for fruitful collaboration; and conversely to recognise when the task at hand is better pursued alone. Research on flexible cooperation in machines has left this meta-level problem largely unexplored, despite its importance for successful collaboration in heterogeneous open-ended environments. Here, we extend the typical Ad Hoc Teamwork (AHT) setting to incorporate the idea of agents having heterogeneous goals that in any given scenario may or may not overlap. We introduce a novel approach to learning policies in this setting, based on a hierarchical combination of imitation and reinforcement learning, and show that it outperforms baseline methods across extended versions of two cooperative environments. We also investigate the contribution of an auxiliary component that learns to model teammates by predicting their actions, finding that its effect on performance is inversely related to the amount of observable information about teammate goals.