DBCal: Density Based Calibration of classifier predictions for uncertainty quantification
This addresses the need for reliable uncertainty measurement in machine learning predictions across scientific domains, though it appears incremental as it builds on existing calibration techniques.
The paper tackles the problem of quantifying uncertainty in classifier predictions by introducing DBCal, a method that accounts for both classifier belief and performance. It achieves an expected calibration error of less than 0.2% on a binary classifier and less than 3% on a semantic segmentation network with extreme class imbalance.
Measurement of uncertainty of predictions from machine learning methods is important across scientific domains and applications. We present, to our knowledge, the first such technique that quantifies the uncertainty of predictions from a classifier and accounts for both the classifier's belief and performance. We prove that our method provides an accurate estimate of the probability that the outputs of two neural networks are correct by showing an expected calibration error of less than 0.2% on a binary classifier, and less than 3% on a semantic segmentation network with extreme class imbalance. We empirically show that the uncertainty returned by our method is an accurate measurement of the probability that the classifier's prediction is correct and, therefore has broad utility in uncertainty propagation.