Heterogeneous Agent Collaborative Reinforcement Learning
This addresses the problem of sample inefficiency and lack of mutual learning in heterogeneous agent systems for reinforcement learning practitioners, offering a novel paradigm with incremental algorithmic improvements.
The paper tackles the inefficiency of isolated on-policy optimization in reinforcement learning by introducing HACRL, a paradigm for collaborative optimization with independent execution, resulting in HACPO improving all agents by an average of 3.3% over GSPO while using half the rollout cost.
We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new learning paradigm that addresses the inefficiencies of isolated on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional teacher-to-student transfer. Building on this paradigm, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation and optimization correctness. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO by an average of 3.3\% while using only half the rollout cost.