Robustness Quantification and Uncertainty Quantification: Comparing Two Methods for Assessing the Reliability of Classifier Predictions
This work addresses reliability assessment for classifier predictions, which is incremental as it compares and combines existing approaches.
The paper tackled the problem of assessing classifier prediction reliability by comparing Robustness Quantification (RQ) and Uncertainty Quantification (UQ), finding that RQ outperforms UQ in standard and distribution shift settings, and a combination of both yields even better results.
We consider two approaches for assessing the reliability of the individual predictions of a classifier: Robustness Quantification (RQ) and Uncertainty Quantification (UQ). We explain the conceptual differences between the two approaches, compare both approaches on a number of benchmark datasets and show that RQ is capable of outperforming UQ, both in a standard setting and in the presence of distribution shift. Beside showing that RQ can be competitive with UQ, we also demonstrate the complementarity of RQ and UQ by showing that a combination of both approaches can lead to even better reliability assessments.