Density estimation in representation space to predict model uncertainty
This addresses the issue of model reliability for users in safety-critical applications like autonomous systems, though it is incremental as it builds on existing uncertainty estimation techniques.
The paper tackles the problem of deep learning models making incorrect predictions with high confidence on unfamiliar test data by proposing a method to estimate prediction uncertainty based on training data density in representation space, achieving state-of-the-art performance in image classification for both in-distribution uncertainty estimation and out-of-distribution detection.
Deep learning models frequently make incorrect predictions with high confidence when presented with test examples that are not well represented in their training dataset. We propose a novel and straightforward approach to estimate prediction uncertainty in a pre-trained neural network model. Our method estimates the training data density in representation space for a novel input. A neural network model then uses this information to determine whether we expect the pre-trained model to make a correct prediction. This uncertainty model is trained by predicting in-distribution errors, but can detect out-of-distribution data without having seen any such example. We test our method for a state-of-the art image classification model in the settings of both in-distribution uncertainty estimation as well as out-of-distribution detection.