An Empirical Study on Ensemble-Based Transfer Learning Bayesian Optimisation with Mixed Variable Types
This work addresses the problem of optimizing expensive black-box functions for researchers and practitioners in machine learning, but it is incremental as it builds on existing transfer learning methods.
The study tackled improving Bayesian optimisation performance through ensemble-based transfer learning methods, finding that warm start initialisation and positive weight constraints in ensemble models generally enhance results, with specific contributions including new pipeline components and benchmarks.
Bayesian optimisation is a sample efficient method for finding a global optimum of expensive black-box objective functions. Historic datasets from related problems can be exploited to help improve performance of Bayesian optimisation by adapting transfer learning methods to various components of the Bayesian optimisation pipeline. In this study we perform an empirical analysis of various ensemble-based transfer learning Bayesian optimisation methods and pipeline components. We expand on previous work in the literature by contributing some specific pipeline components, and three new real-time transfer learning Bayesian optimisation benchmarks. In particular we propose to use a weighting strategy for ensemble surrogate model predictions based on regularised regression with weights constrained to be positive, and a related component for handling the case when transfer learning is not improving Bayesian optimisation performance. We find that in general, two components that help improve transfer learning Bayesian optimisation performance are warm start initialisation and constraining weights used with ensemble surrogate model to be positive.