Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization
This work addresses the gap in using LLA for Bayesian optimization, which is incremental as it extends an existing method to a new domain.
The study investigated the application of the linearized-Laplace approximation (LLA) to Bayesian optimization, a sequential decision-making problem, and found that it demonstrates strong performance and flexibility, though it also identified pitfalls such as issues in unbounded search spaces.
The linearized-Laplace approximation (LLA) has been shown to be effective and efficient in constructing Bayesian neural networks. It is theoretically compelling since it can be seen as a Gaussian process posterior with the mean function given by the neural network's maximum-a-posteriori predictive function and the covariance function induced by the empirical neural tangent kernel. However, while its efficacy has been studied in large-scale tasks like image classification, it has not been studied in sequential decision-making problems like Bayesian optimization where Gaussian processes -- with simple mean functions and kernels such as the radial basis function -- are the de-facto surrogate models. In this work, we study the usefulness of the LLA in Bayesian optimization and highlight its strong performance and flexibility. However, we also present some pitfalls that might arise and a potential problem with the LLA when the search space is unbounded.