We Still Don't Understand High-Dimensional Bayesian Optimization
This work addresses the problem of inefficient high-dimensional optimization for researchers and practitioners, revealing that complex methods are outperformed by simpler ones, which is an incremental but impactful insight.
The paper tackled the challenge of Bayesian optimization in high-dimensional spaces by showing that a simple Bayesian linear regression with a geometric transformation matches state-of-the-art performance on tasks with 60 to 6,000 dimensions, scaling linearly with data to handle over 20,000 observations in molecular optimization.
High-dimensional spaces have challenged Bayesian optimization (BO). Existing methods aim to overcome this so-called curse of dimensionality by carefully encoding structural assumptions, from locality to sparsity to smoothness, into the optimization procedure. Surprisingly, we demonstrate that these approaches are outperformed by arguably the simplest method imaginable: Bayesian linear regression. After applying a geometric transformation to avoid boundary-seeking behavior, Gaussian processes with linear kernels match state-of-the-art performance on tasks with 60- to 6,000-dimensional search spaces. Linear models offer numerous advantages over their non-parametric counterparts: they afford closed-form sampling and their computation scales linearly with data, a fact we exploit on molecular optimization tasks with > 20,000 observations. Coupled with empirical analyses, our results suggest the need to depart from past intuitions about BO methods in high-dimensional spaces.