Parametric $ρ$-Norm Scaling Calibration
This work addresses the problem of uncertainty calibration for machine learning models, particularly in scenarios with limited data, but it is incremental as it builds on existing post-processing calibration methods.
The paper tackles the challenge of calibrating model confidence in limited data sets by introducing a post-processing parametric calibration method called $\rho$-Norm Scaling, which mitigates overconfidence while preserving accuracy, and experimental results show substantial enhancement in uncertainty calibration.
Output uncertainty indicates whether the probabilistic properties reflect objective characteristics of the model output. Unlike most loss functions and metrics in machine learning, uncertainty pertains to individual samples, but validating it on individual samples is unfeasible. When validated collectively, it cannot fully represent individual sample properties, posing a challenge in calibrating model confidence in a limited data set. Hence, it is crucial to consider confidence calibration characteristics. To counter the adverse effects of the gradual amplification of the classifier output amplitude in supervised learning, we introduce a post-processing parametric calibration method, $ρ$-Norm Scaling, which expands the calibrator expression and mitigates overconfidence due to excessive amplitude while preserving accuracy. Moreover, bin-level objective-based calibrator optimization often results in the loss of significant instance-level information. Therefore, we include probability distribution regularization, which incorporates specific priori information that the instance-level uncertainty distribution after calibration should resemble the distribution before calibration. Experimental results demonstrate the substantial enhancement in the post-processing calibrator for uncertainty calibration with our proposed method.