Confidence from Invariance to Image Transformations
This provides a practical solution for improving reliability in computer vision systems, though it is an incremental advancement in confidence estimation methods.
The paper tackles the problem of automatically detecting classification errors in pre-trained visual classifiers by analyzing the invariance of classifier decisions under various image transformations, achieving new state-of-the-art results on error detection and novelty detection tasks across multiple datasets including STL-10, CIFAR-100, and ImageNet.
We develop a technique for automatically detecting the classification errors of a pre-trained visual classifier. Our method is agnostic to the form of the classifier, requiring access only to classifier responses to a set of inputs. We train a parametric binary classifier (error/correct) on a representation derived from a set of classifier responses generated from multiple copies of the same input, each subject to a different natural image transformation. Thus, we establish a measure of confidence in classifier's decision by analyzing the invariance of its decision under various transformations. In experiments with multiple data sets (STL-10,CIFAR-100,ImageNet) and classifiers, we demonstrate new state of the art for the error detection task. In addition, we apply our technique to novelty detection scenarios, where we also demonstrate state of the art results.