Multi-level Training and Bayesian Optimization for Economical Hyperparameter Optimization
This work addresses the problem of time-consuming hyperparameter tuning for machine learning practitioners, presenting an incremental improvement over existing methods.
The paper tackles the challenge of reducing training time in hyperparameter optimization by proposing a multi-level training approach with heavy and light training phases, using a truncated additive Gaussian process model to calibrate performance measurements, and demonstrates competitive results on synthetic examples and various neural networks.
Hyperparameters play a critical role in the performances of many machine learning methods. Determining their best settings or Hyperparameter Optimization (HPO) faces difficulties presented by the large number of hyperparameters as well as the excessive training time. In this paper, we develop an effective approach to reducing the total amount of required training time for HPO. In the initialization, the nested Latin hypercube design is used to select hyperparameter configurations for two types of training, which are, respectively, heavy training and light training. We propose a truncated additive Gaussian process model to calibrate approximate performance measurements generated by light training, using accurate performance measurements generated by heavy training. Based on the model, a sequential model-based algorithm is developed to generate the performance profile of the configuration space as well as find optimal ones. Our proposed approach demonstrates competitive performance when applied to optimize synthetic examples, support vector machines, fully connected networks and convolutional neural networks.