Multi-objective hyperparameter optimization with performance uncertainty
This work addresses hyperparameter optimization for machine learning practitioners by accounting for performance uncertainty, though it is incremental as it builds on established methods like TPE and GPR.
The paper tackles multi-objective hyperparameter optimization by incorporating uncertainty in performance measurements, combining Tree-structured Parzen Estimators with Gaussian Process Regression to improve over existing methods, as shown by gains in hypervolume indicator on test functions and ML problems.
The performance of any Machine Learning (ML) algorithm is impacted by the choice of its hyperparameters. As training and evaluating a ML algorithm is usually expensive, the hyperparameter optimization (HPO) method needs to be computationally efficient to be useful in practice. Most of the existing approaches on multi-objective HPO use evolutionary strategies and metamodel-based optimization. However, few methods have been developed to account for uncertainty in the performance measurements. This paper presents results on multi-objective hyperparameter optimization with uncertainty on the evaluation of ML algorithms. We combine the sampling strategy of Tree-structured Parzen Estimators (TPE) with the metamodel obtained after training a Gaussian Process Regression (GPR) with heterogeneous noise. Experimental results on three analytical test functions and three ML problems show the improvement over multi-objective TPE and GPR, achieved with respect to the hypervolume indicator.