Hyperboost: Hyperparameter Optimization by Gradient Boosting surrogate models
This work addresses the problem of efficiently tuning algorithm hyperparameters for machine learning practitioners, offering an incremental improvement over existing Bayesian Optimization methods.
This paper proposes a new surrogate model for hyperparameter optimization based on gradient boosting, utilizing quantile regression for optimistic performance estimates and a distance metric for exploration. The method empirically outperforms some state-of-the-art techniques on a set of classification problems.
Bayesian Optimization is a popular tool for tuning algorithms in automatic machine learning (AutoML) systems. Current state-of-the-art methods leverage Random Forests or Gaussian processes to build a surrogate model that predicts algorithm performance given a certain set of hyperparameter settings. In this paper, we propose a new surrogate model based on gradient boosting, where we use quantile regression to provide optimistic estimates of the performance of an unobserved hyperparameter setting, and combine this with a distance metric between unobserved and observed hyperparameter settings to help regulate exploration. We demonstrate empirically that the new method is able to outperform some state-of-the art techniques across a reasonable sized set of classification problems.