Can bin-wise scaling improve consistency and adaptivity of prediction uncertainty for machine learning regression ?
This work addresses uncertainty calibration for regression models, which is important for applications like computational chemistry, but it appears incremental as it builds on existing BVS methods.
The paper tackled the problem of improving prediction uncertainty calibration in machine learning regression by exploring adaptations of Binwise Variance Scaling (BVS) with alternative loss functions and input-feature-based binning to enhance adaptivity. The result showed that BVS and its variants were tested on a benchmark dataset for atomization energy prediction and compared to isotonic regression, but no concrete numbers were provided in the abstract.
Binwise Variance Scaling (BVS) has recently been proposed as a post hoc recalibration method for prediction uncertainties of machine learning regression problems that is able of more efficient corrections than uniform variance (or temperature) scaling. The original version of BVS uses uncertainty-based binning, which is aimed to improve calibration conditionally on uncertainty, i.e. consistency. I explore here several adaptations of BVS, in particular with alternative loss functions and a binning scheme based on an input-feature (X) in order to improve adaptivity, i.e. calibration conditional on X. The performances of BVS and its proposed variants are tested on a benchmark dataset for the prediction of atomization energies and compared to the results of isotonic regression.