Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimization
This work addresses a benchmarking gap for researchers in Bayesian optimization, though it is incremental as it focuses on evaluation rather than a new core method.
The paper tackles the lack of systematic benchmarking in Mixed-variable and Combinatorial Bayesian Optimization (MCBO) by introducing a modular framework that enables easy combination of components and provides diverse synthetic and real-world tasks. They implemented 47 novel MCBO algorithms, benchmarked them against 12 existing solvers on 10 tasks with over 4000 experiments, finding a superior combination of primitives that outperforms existing approaches.
This paper introduces a modular framework for Mixed-variable and Combinatorial Bayesian Optimization (MCBO) to address the lack of systematic benchmarking and standardized evaluation in the field. Current MCBO papers often introduce non-diverse or non-standard benchmarks to evaluate their methods, impeding the proper assessment of different MCBO primitives and their combinations. Additionally, papers introducing a solution for a single MCBO primitive often omit benchmarking against baselines that utilize the same methods for the remaining primitives. This omission is primarily due to the significant implementation overhead involved, resulting in a lack of controlled assessments and an inability to showcase the merits of a contribution effectively. To overcome these challenges, our proposed framework enables an effortless combination of Bayesian Optimization components, and provides a diverse set of synthetic and real-world benchmarking tasks. Leveraging this flexibility, we implement 47 novel MCBO algorithms and benchmark them against seven existing MCBO solvers and five standard black-box optimization algorithms on ten tasks, conducting over 4000 experiments. Our findings reveal a superior combination of MCBO primitives outperforming existing approaches and illustrate the significance of model fit and the use of a trust region. We make our MCBO library available under the MIT license at \url{https://github.com/huawei-noah/HEBO/tree/master/MCBO}.