Out-Of-The-Loop Multi-Fidelity Bayesian Optimization
This work is significant for practitioners in fields like molecular optimization and hyperparameter tuning where high-fidelity evaluations are prohibitively expensive, offering a way to leverage existing gold standard data to improve optimization efficiency.
This paper addresses the suboptimality of standard Multi-Fidelity Bayesian Optimization (MF-BO) when the highest-fidelity function is too expensive to be included in the optimization loop. It proposes a method to incorporate historical high-fidelity data, along with task descriptors, to improve performance in such scenarios, demonstrating effectiveness on synthetic functions, chemistry problems, and hyperparameter optimization.
Black-box optimization is a ubiquitous problem in science and engineering, often dealing with expensive objective functions with cheaper lower-fidelity proxies available. Multi-fidelity Bayesian optimization (MF-BO) is a principled approach to this problem, leveraging correlations across different fidelities when querying the objective. However, for many important MF-BO tasks, the true highest-fidelity function is prohibitively expensive to be part of the optimization loop. Nevertheless, practitioners often have gold standard data (observations of the highest-fidelity function) obtained from previous experiments that might provide information for the current task. For instance, in molecular optimization, chemists often pick the top-$k$ candidate molecules using various computer simulations, and later reveal their true objective function values. In this work, we demonstrate the suboptimality of standard MF-BO algorithms in the real-world scenarios above, even under ideal assumptions. Next, we mitigate this problem by incorporating historical high-fidelity data accompanied by task descriptors---which can be explicitly given or extracted from unstructured metadata. We demonstrate the effectiveness of our methods on synthetic functions, as well as real-world problems in chemistry and hyperparameter optimization.