Iterative Deepening Hyperband
This work addresses a specific bottleneck in hyperparameter optimization for machine learning practitioners, offering an incremental improvement over Hyperband.
The paper tackled the problem of Hyperband's non-incremental nature, which forces costly re-runs when the maximal budget is underestimated, by proposing incremental variants that preserve accumulated knowledge and demonstrated their practical utility on benchmark datasets.
Hyperparameter optimization (HPO) is concerned with the automated search for the most appropriate hyperparameter configuration (HPC) of a parameterized machine learning algorithm. A state-of-the-art HPO method is Hyperband, which, however, has its own parameters that influence its performance. One of these parameters, the maximal budget, is especially problematic: If chosen too small, the budget needs to be increased in hindsight and, as Hyperband is not incremental by design, the entire algorithm must be re-run. This is not only costly but also comes with a loss of valuable knowledge already accumulated. In this paper, we propose incremental variants of Hyperband that eliminate these drawbacks, and show that these variants satisfy theoretical guarantees qualitatively similar to those for the original Hyperband with the "right" budget. Moreover, we demonstrate their practical utility in experiments with benchmark data sets.