SLOVA: Uncertainty Estimation Using Single Label One-Vs-All Classifier
This addresses uncertainty estimation for high-risk AI applications, but it is an incremental improvement as it builds on existing one-vs-all methods with fine-tuning.
The paper tackled the problem of unreliable predictive confidence in deep neural networks by introducing the SLOVA model, which redefines one-vs-all probabilities for single-label classification to reduce overconfidence and detect out-of-distribution samples, achieving competitive state-of-the-art performance on in-distribution calibration, robustness under dataset shifts, and excellent out-of-distribution detection.
Deep neural networks present impressive performance, yet they cannot reliably estimate their predictive confidence, limiting their applicability in high-risk domains. We show that applying a multi-label one-vs-all loss reveals classification ambiguity and reduces model overconfidence. The introduced SLOVA (Single Label One-Vs-All) model redefines typical one-vs-all predictive probabilities to a single label situation, where only one class is the correct answer. The proposed classifier is confident only if a single class has a high probability and other probabilities are negligible. Unlike the typical softmax function, SLOVA naturally detects out-of-distribution samples if the probabilities of all other classes are small. The model is additionally fine-tuned with exponential calibration, which allows us to precisely align the confidence score with model accuracy. We verify our approach on three tasks. First, we demonstrate that SLOVA is competitive with the state-of-the-art on in-distribution calibration. Second, the performance of SLOVA is robust under dataset shifts. Finally, our approach performs extremely well in the detection of out-of-distribution samples. Consequently, SLOVA is a tool that can be used in various applications where uncertainty modeling is required.