Asynchronous Policy Gradient Aggregation for Efficient Distributed Reinforcement Learning
This work addresses efficiency bottlenecks in distributed RL for applications requiring scalable and robust learning, though it appears incremental as it builds on existing policy gradient methods.
The paper tackles the problem of distributed reinforcement learning with policy gradient methods under asynchronous and heterogeneous conditions, introducing two algorithms that achieve state-of-the-art efficiency with improved theoretical guarantees and experimental performance.
We study distributed reinforcement learning (RL) with policy gradient methods under asynchronous and parallel computations and communications. While non-distributed methods are well understood theoretically and have achieved remarkable empirical success, their distributed counterparts remain less explored, particularly in the presence of heterogeneous asynchronous computations and communication bottlenecks. We introduce two new algorithms, Rennala NIGT and Malenia NIGT, which implement asynchronous policy gradient aggregation and achieve state-of-the-art efficiency. In the homogeneous setting, Rennala NIGT provably improves the total computational and communication complexity while supporting the AllReduce operation. In the heterogeneous setting, Malenia NIGT simultaneously handles asynchronous computations and heterogeneous environments with strictly better theoretical guarantees. Our results are further corroborated by experiments, showing that our methods significantly outperform prior approaches.