The Bayesian Confidence (BACON) Estimator for Deep Neural Networks
This work addresses calibration issues in deep learning for practitioners, but it is incremental as it builds on existing geometric methods.
The paper tackles the problem of extreme predictions in deep neural networks by introducing the Bayesian Confidence Estimator (BACON), which uses a geometric model and Bayes' Rule with validation data to estimate probabilities, resulting in superior ECE and ACE calibration error compared to Softmax for ResNet-18 and EfficientNet-B0 on CIFAR-10 with an imbalanced test set.
This paper introduces the Bayesian Confidence Estimator (BACON) for deep neural networks. Current practice of interpreting Softmax values in the output layer as probabilities of outcomes is prone to extreme predictions of class probability. In this work we extend Waagen's method of representing the terminal layers with a geometric model, where the probability associated with an output vector is estimated with Bayes' Rule using validation data to provide likelihood and normalization values. This estimator provides superior ECE and ACE calibration error compared to Softmax for ResNet-18 at 85% network accuracy, and EfficientNet-B0 at 95% network accuracy, on the CIFAR-10 dataset with an imbalanced test set, except for very high accuracy edge cases. In addition, when using the ACE metric, BACON demonstrated improved calibration error when estimating probabilities for the imbalanced test set when using actual class distribution fractions.