The case for fully Bayesian optimisation in small-sample trials
This work targets researchers using Bayesian optimization in resource-limited settings, offering an incremental improvement to enhance reliability and prevent wasted resources.
The paper addresses the failure of standard Bayesian optimization using type II maximum likelihood in small-sample trials and advocates for fully Bayesian optimization as a more robust alternative, showing it is practical with minimal performance cost.
While sample efficiency is the main motive for use of Bayesian optimisation when black-box functions are expensive to evaluate, the standard approach based on type II maximum likelihood (ML-II) may fail and result in disappointing performance in small-sample trials. The paper provides three compelling reasons to adopt fully Bayesian optimisation (FBO) as an alternative. First, failures of ML-II are more commonplace than implied by the existing studies using the contrived settings. Second, FBO is more robust than ML-II, and the price of robustness is almost trivial. Third, FBO has become simple to implement and fast enough to be practical. The paper supports the argument using relevant experiments, which reflect the current practice regarding models, algorithms, and software platforms. Since the benefits seem to outweigh the costs, researchers should consider adopting FBO for their applications so that they can guard against potential failures that end up wasting precious research resources.