Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detection
This work identifies a fundamental limitation in current OOD detection methods for safety-critical deep learning applications, impacting practitioners relying on these techniques.
This paper theoretically explains why common uncertainty estimation techniques, particularly those using ReLU networks and softmax activations, fail to reliably detect out-of-distribution (OOD) data in classification tasks. It proves that these methods generalize high confidence to unseen feature space areas, making them unsuitable for OOD detection.
A crucial requirement for reliable deployment of deep learning models for safety-critical applications is the ability to identify out-of-distribution (OOD) data points, samples which differ from the training data and on which a model might underperform. Previous work has attempted to tackle this problem using uncertainty estimation techniques. However, there is empirical evidence that a large family of these techniques do not detect OOD reliably in classification tasks. This paper gives a theoretical explanation for said experimental findings and illustrates it on synthetic data. We prove that such techniques are not able to reliably identify OOD samples in a classification setting, since their level of confidence is generalized to unseen areas of the feature space. This result stems from the interplay between the representation of ReLU networks as piece-wise affine transformations, the saturating nature of activation functions like softmax, and the most widely-used uncertainty metrics.