A Hierarchical Approach to Population Training for Human-AI Collaboration
This work addresses the problem of improving AI robustness in human-AI collaboration for tasks like cooperative gaming, representing an incremental advance over existing population-based training methods.
The paper tackles the challenge of deep reinforcement learning agents collaborating with novel human partners by introducing a hierarchical reinforcement learning method that dynamically switches between multiple best-response policies, demonstrating adaptation to different play styles and skill levels in the Overcooked game environment and validating effectiveness through a human study.
A major challenge for deep reinforcement learning (DRL) agents is to collaborate with novel partners that were not encountered by them during the training phase. This is specifically worsened by an increased variance in action responses when the DRL agents collaborate with human partners due to the lack of consistency in human behaviors. Recent work have shown that training a single agent as the best response to a diverse population of training partners significantly increases an agent's robustness to novel partners. We further enhance the population-based training approach by introducing a Hierarchical Reinforcement Learning (HRL) based method for Human-AI Collaboration. Our agent is able to learn multiple best-response policies as its low-level policy while at the same time, it learns a high-level policy that acts as a manager which allows the agent to dynamically switch between the low-level best-response policies based on its current partner. We demonstrate that our method is able to dynamically adapt to novel partners of different play styles and skill levels in the 2-player collaborative Overcooked game environment. We also conducted a human study in the same environment to test the effectiveness of our method when partnering with real human subjects.