From Solo to Symphony: Orchestrating Multi-Agent Collaboration with Single-Agent Demos
This addresses the bottleneck of costly multi-agent data in scenarios like collaborative coding and search-and-rescue, making cooperative learning more practical, though it is an incremental improvement over existing methods.
The paper tackles the inefficiency of training multi-agent teams from scratch in reinforcement learning by proposing Solo-to-Collaborative RL (SoCo), a framework that transfers knowledge from single-agent demonstrations to improve cooperative learning, resulting in significant boosts in training efficiency and performance across diverse tasks.
Training a team of agents from scratch in multi-agent reinforcement learning (MARL) is highly inefficient, much like asking beginners to play a symphony together without first practicing solo. Existing methods, such as offline or transferable MARL, can ease this burden, but they still rely on costly multi-agent data, which often becomes the bottleneck. In contrast, solo experiences are far easier to obtain in many important scenarios, e.g., collaborative coding, household cooperation, and search-and-rescue. To unlock their potential, we propose Solo-to-Collaborative RL (SoCo), a framework that transfers solo knowledge into cooperative learning. SoCo first pretrains a shared solo policy from solo demonstrations, then adapts it for cooperation during multi-agent training through a policy fusion mechanism that combines an MoE-like gating selector and an action editor. Experiments across diverse cooperative tasks show that SoCo significantly boosts the training efficiency and performance of backbone algorithms. These results demonstrate that solo demonstrations provide a scalable and effective complement to multi-agent data, making cooperative learning more practical and broadly applicable.