Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Story
This work addresses efficiency bottlenecks in large-scale RL applications by improving parallel data collection, though it is incremental as it builds on existing parallel RL frameworks.
The paper tackles the problem of enhancing data diversity in parallel reinforcement learning agents by maximizing the entropy of collected data, resulting in faster sampling rates and reduced redundancies compared to identical agents.
Parallel data collection has redefined Reinforcement Learning (RL), unlocking unprecedented efficiency and powering breakthroughs in large-scale real-world applications. In this paradigm, $N$ identical agents operate in $N$ replicas of an environment simulator, accelerating data collection by a factor of $N$. A critical question arises: \textit{Does specializing the policies of the parallel agents hold the key to surpass the $N$ factor acceleration?} In this paper, we introduce a novel learning framework that maximizes the entropy of collected data in a parallel setting. Our approach carefully balances the entropy of individual agents with inter-agent diversity, effectively minimizing redundancies. The latter idea is implemented with a centralized policy gradient method, which shows promise when evaluated empirically against systems of identical agents, as well as synergy with batch RL techniques that can exploit data diversity. Finally, we provide an original concentration analysis that shows faster rates for specialized parallel sampling distributions, which supports our methodology and may be of independent interest.