Classification Confidence Estimation with Test-Time Data-Augmentation
This addresses the need for error detection in critical applications like medicine and security, though it is incremental as it revises earlier work.
The paper tackles the problem of detecting false predictions in visual classification by proposing a novel confidence estimation method using semantics-preserving image transformations, achieving state-of-the-art performance on various architectures and datasets like ResNext/ImageNet.
Machine learning plays an increasingly significant role in many aspects of our lives (including medicine, transportation, security, justice and other domains), making the potential consequences of false predictions increasingly devastating. These consequences may be mitigated if we can automatically flag such false predictions and potentially assign them to alternative, more reliable mechanisms, that are possibly more costly and involve human attention. This suggests the task of detecting errors, which we tackle in this paper for the case of visual classification. To this end, we propose a novel approach for classification confidence estimation. We apply a set of semantics-preserving image transformations to the input image, and show how the resulting image sets can be used to estimate confidence in the classifier's prediction. We demonstrate the potential of our approach by extensively evaluating it on a wide variety of classifier architectures and datasets, including ResNext/ImageNet, achieving state of the art performance. This paper constitutes a significant revision of our earlier work in this direction (Bahat & Shakhnarovich, 2018).