ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning
For multi-agent reinforcement learning researchers, ACPO provides a principled method to compute joint policy gradients without value decomposition assumptions, addressing a key bottleneck in CTDE.
ACPO introduces an exact decentralized decomposition of the joint policy gradient in MARL, enabling independent per-agent updates that collectively form a single joint gradient step. It outperforms strong baselines on Multi-Robot Warehouse, SMACv2, and MA-MuJoCo, with performance gains increasing with the number of agents.
Cooperative tasks in Multi-Agent Reinforcement Learning (MARL) require agents to collectively maximize a shared return. Under the Centralized Training with Decentralized Execution (CTDE) paradigm, policy gradients have remained difficult to compute directly. Prior methods largely follow two approaches: independent factorized updates with centralized critics, which lack general joint-improvement guarantees without value decomposition assumptions, or alternating best-response updates, which can converge to suboptimal Nash Equilibria. In this paper, we show the joint policy gradient admits an exact decentralized decomposition of per-agent terms, each formed from per-agent score functions and decentralized critics. Based on this decomposition, we develop Agent-Chained Policy Optimization (ACPO), where actors are trained independently, with their updates together constituting a single step on the joint policy gradient. Central to this result is a serialized view of the simultaneous joint decision in which agents commit actions one at a time, each conditioning on a belief over preceding actions. The belief acts as the coordination mechanism which ties the independent per-agent updates into a joint gradient step. We evaluate ACPO on Multi-Robot Warehouse, SMACv2, and MA-MuJoCo, where it outperforms strong baselines, with the gap widening as the number of agents grows.