Bayesian Optimization via Continual Variational Last Layer Training
This work addresses the challenge of surrogate model selection in Bayesian optimization for tasks with complex correlations, offering an incremental improvement over existing methods.
The paper tackled the problem of kernel selection in Gaussian Processes for Bayesian optimization by proposing a method based on variational Bayesian last layers, which showed competitive performance on various problem types, including those where Bayesian neural networks typically struggle, and matched well-tuned GPs on benchmarks.
Gaussian Processes (GPs) are widely seen as the state-of-the-art surrogate models for Bayesian optimization (BO) due to their ability to model uncertainty and their performance on tasks where correlations are easily captured (such as those defined by Euclidean metrics) and their ability to be efficiently updated online. However, the performance of GPs depends on the choice of kernel, and kernel selection for complex correlation structures is often difficult or must be made bespoke. While Bayesian neural networks (BNNs) are a promising direction for higher capacity surrogate models, they have so far seen limited use due to poor performance on some problem types. In this paper, we propose an approach which shows competitive performance on many problem types, including some that BNNs typically struggle with. We build on variational Bayesian last layers (VBLLs), and connect training of these models to exact conditioning in GPs. We exploit this connection to develop an efficient online training algorithm that interleaves conditioning and optimization. Our findings suggest that VBLL networks significantly outperform GPs and other BNN architectures on tasks with complex input correlations, and match the performance of well-tuned GPs on established benchmark tasks.