Decreasing the Computing Time of Bayesian Optimization using Generalizable Memory Pruning
This addresses the computational bottleneck in BO for researchers and practitioners dealing with high-dimensional or massive datasets, offering a generalizable solution that works with any surrogate model and acquisition function, though it is incremental as it builds on existing BO methods.
The paper tackles the long computing times of Bayesian Optimization (BO) on high-dimensional or large datasets by introducing a generalizable wrapper using memory pruning and bounded optimization, which reduces wall-clock computing times from a polynomially increasing pattern to a non-increasing sawtooth trend without sacrificing convergence performance.
Bayesian optimization (BO) suffers from long computing times when processing highly-dimensional or large data sets. These long computing times are a result of the Gaussian process surrogate model having a polynomial time complexity with the number of experiments. Running BO on high-dimensional or massive data sets becomes intractable due to this time complexity scaling, in turn, hindering experimentation. Alternative surrogate models have been developed to reduce the computing utilization of the BO procedure, however, these methods require mathematical alteration of the inherit surrogate function, pigeonholing use into only that function. In this paper, we demonstrate a generalizable BO wrapper of memory pruning and bounded optimization, capable of being used with any surrogate model and acquisition function. Using this memory pruning approach, we show a decrease in wall-clock computing times per experiment of BO from a polynomially increasing pattern to a sawtooth pattern that has a non-increasing trend without sacrificing convergence performance. Furthermore, we illustrate the generalizability of the approach across two unique data sets, two unique surrogate models, and four unique acquisition functions. All model implementations are run on the MIT Supercloud state-of-the-art computing hardware.