Bayesian Generational Population-Based Training
This addresses the problem of automating hyperparameter and architecture selection in reinforcement learning for researchers and practitioners, though it appears incremental as it builds on existing PBT methods.
The paper tackles the brittleness of RL algorithms to hyperparameters and architectures by introducing Bayesian Generational Population-Based Training, which uses trust-region Bayesian optimization and a generational approach to jointly learn configurations on-the-fly, resulting in large performance gains that significantly outperform tuned baselines.
Reinforcement learning (RL) offers the potential for training generally capable agents that can interact autonomously in the real world. However, one key limitation is the brittleness of RL algorithms to core hyperparameters and network architecture choice. Furthermore, non-stationarities such as evolving training data and increased agent complexity mean that different hyperparameters and architectures may be optimal at different points of training. This motivates AutoRL, a class of methods seeking to automate these design choices. One prominent class of AutoRL methods is Population-Based Training (PBT), which have led to impressive performance in several large scale settings. In this paper, we introduce two new innovations in PBT-style methods. First, we employ trust-region based Bayesian Optimization, enabling full coverage of the high-dimensional mixed hyperparameter search space. Second, we show that using a generational approach, we can also learn both architectures and hyperparameters jointly on-the-fly in a single training run. Leveraging the new highly parallelizable Brax physics engine, we show that these innovations lead to large performance gains, significantly outperforming the tuned baseline while learning entire configurations on the fly. Code is available at https://github.com/xingchenwan/bgpbt.