Efficient Nearest Neighbor based Uncertainty Estimation for Natural Language Processing Tasks
This work addresses uncertainty estimation for safety-critical NLP applications, but it is incremental as it builds on existing nearest neighbor and density-based methods.
The paper tackles the problem of uncertainty estimation in deep neural networks for NLP tasks by proposing k-Nearest Neighbor Uncertainty Estimation (kNN-UE), which uses distances and label ratios from neighbors, and shows it outperforms baselines and recent methods in calibration and uncertainty metrics across sentiment analysis, natural language inference, and named entity recognition.
Trustworthiness in model predictions is crucial for safety-critical applications in the real world. However, deep neural networks often suffer from the issues of uncertainty estimation, such as miscalibration. In this study, we propose $k$-Nearest Neighbor Uncertainty Estimation ($k$NN-UE), which is a new uncertainty estimation method that uses not only the distances from the neighbors, but also the ratio of labels in the neighbors. Experiments on sentiment analysis, natural language inference, and named entity recognition show that our proposed method outperforms the baselines and recent density-based methods in several calibration and uncertainty metrics. Moreover, our analyses indicate that approximate nearest neighbor search techniques reduce the inference overhead without significantly degrading the uncertainty estimation performance when they are appropriately combined.