Enhancing Performance and Calibration in Quantile Hyperparameter Optimization
This work addresses hyperparameter optimization for machine learning practitioners, but it is incremental as it builds upon early work in quantile regression.
The paper tackled the problem of Bayesian hyperparameter optimization underperforming in categorical environments or when normality assumptions are violated, by enhancing quantile regression with conformalization and addressing feedback covariate shift. The result showed superior performance to existing quantile methods and validated improved calibration and search performance through rigorous benchmarking against state-of-the-art methods like GP, TPE, and SMAC.
Bayesian hyperparameter optimization relies heavily on Gaussian Process (GP) surrogates, due to robust distributional posteriors and strong performance on limited training samples. GPs however underperform in categorical hyperparameter environments or when assumptions of normality, heteroskedasticity and symmetry are excessively challenged. Conformalized quantile regression can address these estimation weaknesses, while still providing robust calibration guarantees. This study builds upon early work in this area by addressing feedback covariate shift in sequential acquisition and integrating a wider range of surrogate architectures and acquisition functions. Proposed algorithms are rigorously benchmarked against a range of state of the art hyperparameter optimization methods (GP, TPE and SMAC). Findings identify quantile surrogate architectures and acquisition functions yielding superior performance to the current quantile literature, while validating the beneficial impact of conformalization on calibration and search performance.