Risk-Controlling Model Selection via Guided Bayesian Optimization
This addresses the need for reliable model selection under multiple constraints for practitioners, though it is incremental as it builds on existing Bayesian Optimization and risk control methods.
The paper tackles the problem of selecting machine learning model configurations that meet user-specified risk limits while optimizing other metrics, by combining Bayesian Optimization with risk-controlling procedures to guarantee risk levels and demonstrate effectiveness across tasks like error rates, fairness, and computational costs.
Adjustable hyperparameters of machine learning models typically impact various key trade-offs such as accuracy, fairness, robustness, or inference cost. Our goal in this paper is to find a configuration that adheres to user-specified limits on certain risks while being useful with respect to other conflicting metrics. We solve this by combining Bayesian Optimization (BO) with rigorous risk-controlling procedures, where our core idea is to steer BO towards an efficient testing strategy. Our BO method identifies a set of Pareto optimal configurations residing in a designated region of interest. The resulting candidates are statistically verified and the best-performing configuration is selected with guaranteed risk levels. We demonstrate the effectiveness of our approach on a range of tasks with multiple desiderata, including low error rates, equitable predictions, handling spurious correlations, managing rate and distortion in generative models, and reducing computational costs.