Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameters
This work addresses a practical issue for automated machine learning practitioners, but it is incremental as it builds on existing Bayesian Optimization methods.
The paper tackles the problem of neglecting the tuning of Bayesian Optimization's own hyperparameters, showing empirically that such tuning improves any-time performance across various benchmarks, with optimized settings transferring well to similar and some other problem families.
Bayesian Optimization (BO) is a common approach for hyperparameter optimization (HPO) in automated machine learning. Although it is well-accepted that HPO is crucial to obtain well-performing machine learning models, tuning BO's own hyperparameters is often neglected. In this paper, we empirically study the impact of optimizing BO's own hyperparameters and the transferability of the found settings using a wide range of benchmarks, including artificial functions, HPO and HPO combined with neural architecture search. In particular, we show (i) that tuning can improve the any-time performance of different BO approaches, that optimized BO settings also perform well (ii) on similar problems and (iii) partially even on problems from other problem families, and (iv) which BO hyperparameters are most important.