BOOST: Bayesian Optimization with Optimal Kernel and Acquisition Function Selection Technique
This addresses a practical problem for practitioners using Bayesian optimization in expensive black-box problems, offering an automated solution to improve efficiency, though it is incremental as it builds on existing kernel and acquisition function improvements.
The paper tackles the challenge of selecting optimal kernel and acquisition function pairs in Bayesian optimization, which is critical for performance but often done manually or heuristically, by proposing BOOST, a framework that automates this selection through offline evaluation and consistently outperforms standard and adaptive methods in experiments.
The performance of Bayesian optimization (BO), a highly sample-efficient method for expensive black-box problems, is critically governed by the selection of its hyperparameters, including the kernel and acquisition functions. This presents a significant practical challenge: an inappropriate combination can lead to poor performance and wasted evaluations. While individual improvements to kernel functions (e.g., tree-based kernels, deep kernel learning) and acquisition functions (e.g., multi-step lookahead, tree-based planning) have been actively explored, the joint and autonomous selection of the best pair has been largely overlooked, forcing practitioners to rely on heuristics or costly manual tuning. We propose BOOST (Bayesian Optimization with Optimal Kernel and Acquisition Function Selection Technique), a novel framework that automates this selection. BOOST utilizes a lightweight, offline evaluation stage to predict the performance of various kernel-acquisition pairs and identify the most promising pair before committing to expensive evaluations. Using K-means clustering, BOOST first selects initial subsets from previously observed data-in-hand and prepares all possible kernel-acquisition pairs from user-chosen candidates. For each pair, BOOST conducts internal BO runs starting with the initial subset, evaluating how many iterations are required to find the target value within the remaining data, thereby identifying the pair with the best retrospective performance for future optimization. Experiments on synthetic benchmarks and real-world hyperparameter optimization tasks demonstrate that BOOST consistently outperforms standard BO with fixed hyperparameters and state-of-the-art adaptive methods, highlighting its effectiveness and robustness in diverse problem landscapes.