Trust Region Bounds for Decentralized PPO Under Non-stationarity
This provides theoretical foundations for practical MARL methods, though it is incremental as it analyzes existing approaches rather than proposing new ones.
The paper tackles the problem of theoretical understanding for decentralized policy optimization in multi-agent reinforcement learning under non-stationary dynamics, showing that trust region constraints provide monotonic improvement guarantees and explaining the strong performance of existing methods like IPPO and MAPPO.
We present trust region bounds for optimizing decentralized policies in cooperative Multi-Agent Reinforcement Learning (MARL), which holds even when the transition dynamics are non-stationary. This new analysis provides a theoretical understanding of the strong performance of two recent actor-critic methods for MARL, which both rely on independent ratios, i.e., computing probability ratios separately for each agent's policy. We show that, despite the non-stationarity that independent ratios cause, a monotonic improvement guarantee still arises as a result of enforcing the trust region constraint over all decentralized policies. We also show this trust region constraint can be effectively enforced in a principled way by bounding independent ratios based on the number of agents in training, providing a theoretical foundation for proximal ratio clipping. Finally, our empirical results support the hypothesis that the strong performance of IPPO and MAPPO is a direct result of enforcing such a trust region constraint via clipping in centralized training, and tuning the hyperparameters with regards to the number of agents, as predicted by our theoretical analysis.