Are Random Decompositions all we need in High Dimensional Bayesian Optimisation?
This work addresses the problem of high-dimensional Bayesian optimization for researchers and practitioners, offering a plug-and-play method that improves performance in complex tasks.
The paper tackles the challenge of scaling Bayesian optimization to high-dimensional problems by investigating data-independent decomposition sampling rules, finding that random tree-based decompositions offer theoretical guarantees and lead to the RDUCB algorithm, which achieves significant empirical gains over state-of-the-art methods on benchmarks.
Learning decompositions of expensive-to-evaluate black-box functions promises to scale Bayesian optimisation (BO) to high-dimensional problems. However, the success of these techniques depends on finding proper decompositions that accurately represent the black-box. While previous works learn those decompositions based on data, we investigate data-independent decomposition sampling rules in this paper. We find that data-driven learners of decompositions can be easily misled towards local decompositions that do not hold globally across the search space. Then, we formally show that a random tree-based decomposition sampler exhibits favourable theoretical guarantees that effectively trade off maximal information gain and functional mismatch between the actual black-box and its surrogate as provided by the decomposition. Those results motivate the development of the random decomposition upper-confidence bound algorithm (RDUCB) that is straightforward to implement - (almost) plug-and-play - and, surprisingly, yields significant empirical gains compared to the previous state-of-the-art on a comprehensive set of benchmarks. We also confirm the plug-and-play nature of our modelling component by integrating our method with HEBO, showing improved practical gains in the highest dimensional tasks from Bayesmark.