Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimization
This work addresses risk-aware hyperparameter optimization for engineering applications, offering an incremental improvement over prior methods.
The paper tackles the need for robust statistical reliability guarantees in AI model calibration by introducing a variant of the learn-then-test method that provides guarantees on quantiles of risk measures, applied to a radio access scheduling problem.
The increasing adoption of Artificial Intelligence (AI) in engineering problems calls for the development of calibration methods capable of offering robust statistical reliability guarantees. The calibration of black box AI models is carried out via the optimization of hyperparameters dictating architecture, optimization, and/or inference configuration. Prior work has introduced learn-then-test (LTT), a calibration procedure for hyperparameter optimization (HPO) that provides statistical guarantees on average performance measures. Recognizing the importance of controlling risk-aware objectives in engineering contexts, this work introduces a variant of LTT that is designed to provide statistical guarantees on quantiles of a risk measure. We illustrate the practical advantages of this approach by applying the proposed algorithm to a radio access scheduling problem.