Structured Diversification Emergence via Reinforced Organization Control and Hierarchical Consensus Learning
This work aims to improve the efficiency of cooperation and exploration for multi-agent reinforcement learning researchers by providing a structured approach to team formation, which is an incremental improvement over existing heuristic or blackbox methods.
This paper addresses the challenge of forming cooperative teams in multi-agent reinforcement learning (MARL) by proposing Rochico, a framework that learns adaptive grouping policies and hierarchical consensus. Experiments on four large-scale cooperation tasks demonstrate that Rochico significantly outperforms current state-of-the-art algorithms in exploration efficiency and cooperation strength.
When solving a complex task, humans will spontaneously form teams and to complete different parts of the whole task, respectively. Meanwhile, the cooperation between teammates will improve efficiency. However, for current cooperative MARL methods, the cooperation team is constructed through either heuristics or end-to-end blackbox optimization. In order to improve the efficiency of cooperation and exploration, we propose a structured diversification emergence MARL framework named {\sc{Rochico}} based on reinforced organization control and hierarchical consensus learning. {\sc{Rochico}} first learns an adaptive grouping policy through the organization control module, which is established by independent multi-agent reinforcement learning. Further, the hierarchical consensus module based on the hierarchical intentions with consensus constraint is introduced after team formation. Simultaneously, utilizing the hierarchical consensus module and a self-supervised intrinsic reward enhanced decision module, the proposed cooperative MARL algorithm {\sc{Rochico}} can output the final diversified multi-agent cooperative policy. All three modules are organically combined to promote the structured diversification emergence. Comparative experiments on four large-scale cooperation tasks show that {\sc{Rochico}} is significantly better than the current SOTA algorithms in terms of exploration efficiency and cooperation strength.