K-Myriad: Jump-starting reinforcement learning with unsupervised parallel agents
This work addresses the need for more effective parallelization strategies in reinforcement learning, offering a scalable approach to improve exploration and policy initialization, though it appears incremental in advancing existing parallel methods.
The paper tackled the problem of limited exploration diversity in parallel reinforcement learning by proposing K-Myriad, an unsupervised method that maximizes collective state entropy with a population of parallel policies, resulting in higher training efficiency and discovery of heterogeneous solutions in high-dimensional continuous control tasks.
Parallelization in Reinforcement Learning is typically employed to speed up the training of a single policy, where multiple workers collect experience from an identical sampling distribution. This common design limits the potential of parallelization by neglecting the advantages of diverse exploration strategies. We propose K-Myriad, a scalable and unsupervised method that maximizes the collective state entropy induced by a population of parallel policies. By cultivating a portfolio of specialized exploration strategies, K-Myriad provides a robust initialization for Reinforcement Learning, leading to both higher training efficiency and the discovery of heterogeneous solutions. Experiments on high-dimensional continuous control tasks, with large-scale parallelization, demonstrate that K-Myriad can learn a broad set of distinct policies, highlighting its effectiveness for collective exploration and paving the way towards novel parallelization strategies.