Neural Recursive Belief States in Multi-Agent Reinforcement Learning
This work tackles the problem of intractable hierarchical beliefs in multi-agent reinforcement learning, which is a significant challenge for developing more human-like AI agents capable of complex social interactions.
This paper addresses the challenge of multi-agent reinforcement learning where co-player policies depend on private information. The authors propose using recursive deep generative models to approximate hierarchical belief structures, resulting in agents that outperform model-free baselines and agents with lower-order belief models.
In multi-agent reinforcement learning, the problem of learning to act is particularly difficult because the policies of co-players may be heavily conditioned on information only observed by them. On the other hand, humans readily form beliefs about the knowledge possessed by their peers and leverage beliefs to inform decision-making. Such abilities underlie individual success in a wide range of Markov games, from bluffing in Poker to conditional cooperation in the Prisoner's Dilemma, to convention-building in Bridge. Classical methods are usually not applicable to complex domains due to the intractable nature of hierarchical beliefs (i.e. beliefs of other agents' beliefs). We propose a scalable method to approximate these belief structures using recursive deep generative models, and to use the belief models to obtain representations useful to acting in complex tasks. Our agents trained with belief models outperform model-free baselines with equivalent representational capacity using common training paradigms. We also show that higher-order belief models outperform agents with lower-order models.