Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
This addresses the problem of efficient hyperparameter tuning in reinforcement learning for practitioners, though it is incremental as it extends an existing method to handle categorical variables.
The paper tackled the limitation of PB2, a population-based AutoRL algorithm, which only handles continuous hyperparameters, by introducing a new hierarchical approach for optimizing both continuous and categorical variables, achieving improved generalization on the Procgen benchmark.
Despite a series of recent successes in reinforcement learning (RL), many RL algorithms remain sensitive to hyperparameters. As such, there has recently been interest in the field of AutoRL, which seeks to automate design decisions to create more general algorithms. Recent work suggests that population based approaches may be effective AutoRL algorithms, by learning hyperparameter schedules on the fly. In particular, the PB2 algorithm is able to achieve strong performance in RL tasks by formulating online hyperparameter optimization as time varying GP-bandit problem, while also providing theoretical guarantees. However, PB2 is only designed to work for continuous hyperparameters, which severely limits its utility in practice. In this paper we introduce a new (provably) efficient hierarchical approach for optimizing both continuous and categorical variables, using a new time-varying bandit algorithm specifically designed for the population based training regime. We evaluate our approach on the challenging Procgen benchmark, where we show that explicitly modelling dependence between data augmentation and other hyperparameters improves generalization.