Long-run Behaviour of Multi-fidelity Bayesian Optimisation
This work addresses potential reliability issues in MFBO for researchers, but it is incremental as it builds on existing benchmarks and observations.
The paper investigates the long-run behavior of Multi-fidelity Bayesian Optimization (MFBO), finding that it can underperform compared to single-fidelity methods in certain scenarios, which could impact its reliability for research tasks.
Multi-fidelity Bayesian Optimisation (MFBO) has been shown to generally converge faster than single-fidelity Bayesian Optimisation (SFBO) (Poloczek et al. (2017)). Inspired by recent benchmark papers, we are investigating the long-run behaviour of MFBO, based on observations in the literature that it might under-perform in certain scenarios (Mikkola et al. (2023), Eggensperger et al. (2021)). An under-performance of MBFO in the long-run could significantly undermine its application to many research tasks, especially when we are not able to identify when the under-performance begins. We create a simple benchmark study, showcase empirical results and discuss scenarios and possible reasons of under-performance.