Hyperparameter Transfer Learning with Adaptive Complexity
This work addresses a specific bottleneck in hyperparameter tuning for machine learning practitioners, offering an incremental improvement over existing multi-task Bayesian optimization methods.
The paper tackles the mismatch in evaluation counts between current and past tasks in multi-task Bayesian optimization for hyperparameter tuning, proposing a method that learns ordered non-linear basis functions to handle different data regimes, resulting in improved sample efficiency across various tuning problems.
Bayesian optimization (BO) is a sample efficient approach to automatically tune the hyperparameters of machine learning models. In practice, one frequently has to solve similar hyperparameter tuning problems sequentially. For example, one might have to tune a type of neural network learned across a series of different classification problems. Recent work on multi-task BO exploits knowledge gained from previous tuning tasks to speed up a new tuning task. However, previous approaches do not account for the fact that BO is a sequential decision making procedure. Hence, there is in general a mismatch between the number of evaluations collected in the current tuning task compared to the number of evaluations accumulated in all previously completed tasks. In this work, we enable multi-task BO to compensate for this mismatch, such that the transfer learning procedure is able to handle different data regimes in a principled way. We propose a new multi-task BO method that learns a set of ordered, non-linear basis functions of increasing complexity via nested drop-out and automatic relevance determination. Experiments on a variety of hyperparameter tuning problems show that our method improves the sample ef