On double-descent in uncertainty quantification in overparametrized models
This work addresses the problem of unreliable uncertainty estimates in overparametrized models for machine learning practitioners, offering theoretical insights but is incremental as it builds on existing methods and models.
The paper investigates uncertainty quantification in overparametrized neural networks using a random features model, revealing a double-descent behavior in calibration curves for optimally regularized estimators, while empirical Bayes methods remain well-calibrated despite higher generalization error.
Uncertainty quantification is a central challenge in reliable and trustworthy machine learning. Naive measures such as last-layer scores are well-known to yield overconfident estimates in the context of overparametrized neural networks. Several methods, ranging from temperature scaling to different Bayesian treatments of neural networks, have been proposed to mitigate overconfidence, most often supported by the numerical observation that they yield better calibrated uncertainty measures. In this work, we provide a sharp comparison between popular uncertainty measures for binary classification in a mathematically tractable model for overparametrized neural networks: the random features model. We discuss a trade-off between classification accuracy and calibration, unveiling a double descent like behavior in the calibration curve of optimally regularized estimators as a function of overparametrization. This is in contrast with the empirical Bayes method, which we show to be well calibrated in our setting despite the higher generalization error and overparametrization.