A systematic study comparing hyperparameter optimization engines on tabular data
This work provides practical guidance for researchers and practitioners in machine learning on selecting hyperparameter optimization engines, though it is incremental as it builds on existing methods and benchmarks.
The study systematically compared hyperparameter optimization engines on tabular data, finding that most engines outperform random search, with HEBO, AX, and BlendSearch standing out, and revealing that some engines specialize in optimizing specific learning algorithms.
We run an independent comparison of all hyperparameter optimization (hyperopt) engines available in the Ray Tune library. We introduce two ways to normalize and aggregate statistics across data sets and models, one rank-based, and another one sandwiching the score between the random search score and the full grid search score. This affords us i) to rank the hyperopt engines, ii) to make generalized and statistically significant statements on how much they improve over random search, and iii) to make recommendations on which engine should be used to hyperopt a given learning algorithm. We find that most engines beat random search, but that only three of them (HEBO, AX, and BlendSearch) clearly stand out. We also found that some engines seem to specialize in hyperopting certain learning algorithms, which makes it tricky to use hyperopt in comparison studies, since the choice of the hyperopt technique may favor some of the models in the comparison.