Fubiao Zhang

Tianze Zhou, Fubiao Zhang, Kun Shao et al.

Extending transfer learning to cooperative multi-agent reinforcement learning (MARL) has recently received much attention. In contrast to the single-agent setting, the coordination indispensable in cooperative MARL constrains each agent's policy. However, existing transfer methods focus exclusively on agent policy and ignores coordination knowledge. We propose a new architecture that realizes robust coordination knowledge transfer through appropriate decomposition of the overall coordination into several coordination patterns. We use a novel mixing network named level-adaptive QTransformer (LA-QTransformer) to realize agent coordination that considers credit assignment, with appropriate coordination patterns for different agents realized by a novel level-adaptive Transformer (LA-Transformer) dedicated to the transfer of coordination knowledge. In addition, we use a novel agent network named Population Invariant agent with Transformer (PIT) to realize the coordination transfer in more varieties of scenarios. Extensive experiments in StarCraft II micro-management show that LA-QTransformer together with PIT achieves superior performance compared with state-of-the-art baselines.

Tianze Zhou, Fubiao Zhang, Pan Tang et al.

Recent advances have witnessed that value decomposed-based multi-agent reinforcement learning methods make an efficient performance in coordination tasks. Most current methods assume that agents can make communication to assist decisions, which is impractical in some situations. In this paper, we propose a semi-communication method to enable agents can exchange information without communication. Specifically, we introduce a group concept to help agents learning a belief which is a type of consensus. With this consensus, adjacent agents tend to accomplish similar sub-tasks to achieve cooperation. We design a novel agent structure named Belief in Graph Clustering(BGC), composed of an agent characteristic module, a belief module, and a fusion module. To represent each agent characteristic, we use an MLP-based characteristic module to generate agent unique features. Inspired by the neighborhood cognitive consistency, we propose a group-based module to divide adjacent agents into a small group and minimize in-group agents' beliefs to accomplish similar sub-tasks. Finally, we use a hyper-network to merge these features and produce agent actions. To overcome the agent consistent problem brought by GAT, a split loss is introduced to distinguish different agents. Results reveal that the proposed method achieves a significant improvement in the SMAC benchmark. Because of the group concept, our approach maintains excellent performance with an increase in the number of agents.