Statistically Valid Hyperparameter Selection: From Tuning to Guarantees
For practitioners deploying AI systems, this work addresses the lack of statistical guarantees in hyperparameter tuning, enabling provably safe and reliable model selection.
This monograph introduces a unified statistical framework for hyperparameter selection that provides formal finite-sample guarantees on reliability requirements, such as bounds on average or quantile risk, using the learn-then-test paradigm based on multiple hypothesis testing.
Hyperparameter selection is a critical step in the deployment of modern artificial intelligence systems, given the need to tune degrees of freedom such as inference-time parameters, implementation-level settings, and thresholds driving decision rules. Despite its practical importance, hyperparameter selection is typically performed using best-effort empirical methods such as grid search or Bayesian optimization, which provide no formal statistical guarantees on reliability or safety. This monograph presents a unified statistical framework for reliable hyperparameter selection, centered on the learn-then-test (LTT) paradigm, which formulates the problem as multiple hypothesis testing over a candidate set of hyperparameters. The framework enables the selection of hyperparameters that provably satisfy application-specific reliability requirements -- such as bounds on average risk, quantile risk, or information-theoretic constraints -- with explicit, finite-sample control of error probabilities. The supporting statistical machinery, namely p-values, e-values, and concentration inequalities, is developed from first principles in a dedicated appendix.