Searching in the Forest for Local Bayesian Optimization
This is an incremental improvement for researchers and practitioners in machine learning, addressing sample efficiency in HPO for mid-sized configuration spaces.
The paper tackles hyperparameter optimization (HPO) by proposing BOinG, a two-stage Bayesian optimization method that first models the global landscape with a random forest and then focuses on promising local regions, showing strong performance on mid-sized problems from synthetic functions and HPO.
Because of its sample efficiency, Bayesian optimization (BO) has become a popular approach dealing with expensive black-box optimization problems, such as hyperparameter optimization (HPO). Recent empirical experiments showed that the loss landscapes of HPO problems tend to be more benign than previously assumed, i.e. in the best case uni-modal and convex, such that a BO framework could be more efficient if it can focus on those promising local regions. In this paper, we propose BOinG, a two-stage approach that is tailored toward mid-sized configuration spaces, as one encounters in many HPO problems. In the first stage, we build a scalable global surrogate model with a random forest to describe the overall landscape structure. Further, we choose a promising subregion via a bottom-up approach on the upper-level tree structure. In the second stage, a local model in this subregion is utilized to suggest the point to be evaluated next. Empirical experiments show that BOinG is able to exploit the structure of typical HPO problems and performs particularly well on mid-sized problems from synthetic functions and HPO.