A Deeper Understanding of State-Based Critics in Multi-Agent Reinforcement Learning
This work addresses a theoretical gap for researchers in multi-agent reinforcement learning, though it is incremental as it builds on existing centralized training paradigms.
The paper tackles the problem of bias and variance in state-based critics used in centralized training for decentralized execution in multi-agent reinforcement learning, showing that these critics can introduce bias in policy gradient estimates and increase gradient variance, contrary to common intuition, with empirical comparisons on benchmarks detailing environmental effects.
Centralized Training for Decentralized Execution, where training is done in a centralized offline fashion, has become a popular solution paradigm in Multi-Agent Reinforcement Learning. Many such methods take the form of actor-critic with state-based critics, since centralized training allows access to the true system state, which can be useful during training despite not being available at execution time. State-based critics have become a common empirical choice, albeit one which has had limited theoretical justification or analysis. In this paper, we show that state-based critics can introduce bias in the policy gradient estimates, potentially undermining the asymptotic guarantees of the algorithm. We also show that, even if the state-based critics do not introduce any bias, they can still result in a larger gradient variance, contrary to the common intuition. Finally, we show the effects of the theories in practice by comparing different forms of centralized critics on a wide range of common benchmarks, and detail how various environmental properties are related to the effectiveness of different types of critics.