CRUDE: Calibrating Regression Uncertainty Distributions Empirically
This addresses a critical need for reliable uncertainty calibration in fields like autonomous vehicles and medicine, offering a practical solution for regression models.
The paper tackles the problem of uncalibrated and overconfident uncertainty estimates in regression settings by introducing CRUDE, a calibration method that does not assume a specific uncertainty distribution. It demonstrates consistently sharper, better calibrated, and more accurate uncertainty estimates than state-of-the-art techniques across extensive regression tasks.
Calibrated uncertainty estimates in machine learning are crucial to many fields such as autonomous vehicles, medicine, and weather and climate forecasting. While there is extensive literature on uncertainty calibration for classification, the classification findings do not always translate to regression. As a result, modern models for predicting uncertainty in regression settings typically produce uncalibrated and overconfident estimates. To address these gaps, we present a calibration method for regression settings that does not assume a particular uncertainty distribution over the error: Calibrating Regression Uncertainty Distributions Empirically (CRUDE). CRUDE makes the weaker assumption that error distributions have a constant arbitrary shape across the output space, shifted by predicted mean and scaled by predicted standard deviation. We detail a theoretical connection between CRUDE and conformal inference. Across an extensive set of regression tasks, CRUDE demonstrates consistently sharper, better calibrated, and more accurate uncertainty estimates than state-of-the-art techniques.