HyperBO+: Pre-training a universal prior for Bayesian optimization with hierarchical Gaussian processes
This addresses the need for more flexible and generalizable priors in Bayesian optimization, particularly for practitioners dealing with multiple search spaces, though it builds incrementally on prior transfer learning methods.
The paper tackles the problem of Bayesian optimization requiring hand-crafted priors by introducing HyperBO+, a pre-training method for hierarchical Gaussian processes that enables a universal prior to work across functions with different domains, achieving lower regrets on real-world hyperparameter tuning tasks compared to baselines.
Bayesian optimization (BO), while proved highly effective for many black-box function optimization tasks, requires practitioners to carefully select priors that well model their functions of interest. Rather than specifying by hand, researchers have investigated transfer learning based methods to automatically learn the priors, e.g. multi-task BO (Swersky et al., 2013), few-shot BO (Wistuba and Grabocka, 2021) and HyperBO (Wang et al., 2022). However, those prior learning methods typically assume that the input domains are the same for all tasks, weakening their ability to use observations on functions with different domains or generalize the learned priors to BO on different search spaces. In this work, we present HyperBO+: a pre-training approach for hierarchical Gaussian processes that enables the same prior to work universally for Bayesian optimization on functions with different domains. We propose a two-step pre-training method and analyze its appealing asymptotic properties and benefits to BO both theoretically and empirically. On real-world hyperparameter tuning tasks that involve multiple search spaces, we demonstrate that HyperBO+ is able to generalize to unseen search spaces and achieves lower regrets than competitive baselines.