Hybrid Training for Enhanced Multi-task Generalization in Multi-agent Reinforcement Learning
This addresses the challenge of computational waste and limited applicability in MARL for scenarios requiring generalization to diverse tasks, though it appears incremental as it builds on existing online and offline approaches.
The paper tackles the problem of multi-task generalization in multi-agent reinforcement learning (MARL) by introducing HyGen, a hybrid framework that integrates online and offline learning, resulting in outperforming existing methods on the StarCraft multi-agent challenge.
In multi-agent reinforcement learning (MARL), achieving multi-task generalization to diverse agents and objectives presents significant challenges. Existing online MARL algorithms primarily focus on single-task performance, but their lack of multi-task generalization capabilities typically results in substantial computational waste and limited real-life applicability. Meanwhile, existing offline multi-task MARL approaches are heavily dependent on data quality, often resulting in poor performance on unseen tasks. In this paper, we introduce HyGen, a novel hybrid MARL framework, Hybrid Training for Enhanced Multi-Task Generalization, which integrates online and offline learning to ensure both multi-task generalization and training efficiency. Specifically, our framework extracts potential general skills from offline multi-task datasets. We then train policies to select the optimal skills under the centralized training and decentralized execution paradigm (CTDE). During this stage, we utilize a replay buffer that integrates both offline data and online interactions. We empirically demonstrate that our framework effectively extracts and refines general skills, yielding impressive generalization to unseen tasks. Comparative analyses on the StarCraft multi-agent challenge show that HyGen outperforms a wide range of existing solely online and offline methods.