Practical Transfer Learning for Bayesian Optimization
This work addresses the problem of efficient hyperparameter tuning for machine learning practitioners by enabling faster optimization through transfer learning, though it appears incremental as it builds on existing extensions.
The paper tackles hyperparameter optimization across multiple datasets by developing a hyperparameter-free ensemble model for Bayesian optimization that generalizes two existing transfer learning approaches, establishing a worst-case performance bound. Using benchmark problems, it demonstrates substantial reductions in optimization time compared to standard methods and improvements over state-of-the-art transfer hyperparameter optimization.
When hyperparameter optimization of a machine learning algorithm is repeated for multiple datasets it is possible to transfer knowledge to an optimization run on a new dataset. We develop a new hyperparameter-free ensemble model for Bayesian optimization that is a generalization of two existing transfer learning extensions to Bayesian optimization and establish a worst-case bound compared to vanilla Bayesian optimization. Using a large collection of hyperparameter optimization benchmark problems, we demonstrate that our contributions substantially reduce optimization time compared to standard Gaussian process-based Bayesian optimization and improve over the current state-of-the-art for transfer hyperparameter optimization.