Deep Ranking Ensembles for Hyperparameter Optimization
This addresses the problem of automating hyperparameter tuning for machine learning practitioners, representing a significant advancement rather than an incremental improvement.
The paper tackles hyperparameter optimization by proposing a novel method that meta-learns neural network surrogates optimized for ranking configurations' performances with uncertainty modeling via ensembling, achieving new state-of-the-art results across 12 baselines, 16 search spaces, and 86 datasets/tasks.
Automatically optimizing the hyperparameters of Machine Learning algorithms is one of the primary open questions in AI. Existing work in Hyperparameter Optimization (HPO) trains surrogate models for approximating the response surface of hyperparameters as a regression task. In contrast, we hypothesize that the optimal strategy for training surrogates is to preserve the ranks of the performances of hyperparameter configurations as a Learning to Rank problem. As a result, we present a novel method that meta-learns neural network surrogates optimized for ranking the configurations' performances while modeling their uncertainty via ensembling. In a large-scale experimental protocol comprising 12 baselines, 16 HPO search spaces and 86 datasets/tasks, we demonstrate that our method achieves new state-of-the-art results in HPO.