Heteroscedastic Calibration of Uncertainty Estimators in Deep Learning
This addresses the challenge of translating uncertainty estimates into reliable decisions for high-risk applications in deep learning, representing an incremental improvement over existing recalibration methods.
The paper tackled the problem of poorly calibrated uncertainty estimates in deep learning, which hinders actionable insights, by proposing heteroscedastic regression as a surrogate for calibration, eliminating the need for separate recalibration steps and showing effectiveness in regression experiments with two popular estimators.
The role of uncertainty quantification (UQ) in deep learning has become crucial with growing use of predictive models in high-risk applications. Though a large class of methods exists for measuring deep uncertainties, in practice, the resulting estimates are found to be poorly calibrated, thus making it challenging to translate them into actionable insights. A common workaround is to utilize a separate recalibration step, which adjusts the estimates to compensate for the miscalibration. Instead, we propose to repurpose the heteroscedastic regression objective as a surrogate for calibration and enable any existing uncertainty estimator to be inherently calibrated. In addition to eliminating the need for recalibration, this also regularizes the training process. Using regression experiments, we demonstrate the effectiveness of the proposed heteroscedastic calibration with two popular uncertainty estimators.