Counterfactual Credit Policy Optimization for Multi-Agent Collaboration
This addresses the problem of free-riding and inefficient training in multi-agent LLM collaboration, offering a more effective credit assignment method for researchers and practitioners in AI.
The paper tackles the credit assignment problem in collaborative multi-agent large language models, where shared global rewards obscure individual contributions, by introducing Counterfactual Credit Policy Optimization (CCPO), which assigns agent-specific learning signals through counterfactual trajectories and outperforms strong multi-agent RL baselines on mathematical and logical reasoning benchmarks.
Collaborative multi-agent large language models (LLMs) can solve complex reasoning tasks by decomposing roles and aggregating diverse hypotheses. Yet, reinforcement learning (RL) for such systems is often undermined by credit assignment: a shared global reward obscures individual contributions, inflating update variance and encouraging free-riding. We introduce Counterfactual Credit Policy Optimization (CCPO), a framework that assigns agent-specific learning signals by estimating each agent's marginal contribution through counterfactual trajectories. CCPO builds dynamic counterfactual baselines that simulate outcomes with an agent's contribution removed, yielding role-sensitive advantages for policy optimization. To further improve stability under heterogeneous tasks and data distributions, we propose a global-history-aware normalization scheme that calibrates advantages using global rollout statistics. We evaluate CCPO on two collaboration topologies: a sequential Think--Reason dyad and multi-agent voting. Across mathematical and logical reasoning benchmarks, CCPO mitigates free-riding and outperforms strong multi-agent RL baselines, yielding finer-grained and more effective credit assignment for collaborative LLM training. Our code is available at https://github.com/bhai114/ccpo.