Confidence-based Out-of-Distribution Detection: A Comparative Study and Analysis
This work addresses the need for reliable OOD detection in critical applications such as clinical decision making, though it is incremental as it compares existing methods without introducing new ones.
The paper tackled the problem of out-of-distribution (OOD) detection for image classification models in real-world applications, finding that high performance on a computer vision benchmark does not directly translate to accuracy in medical imaging tasks like disease classification with chest X-rays.
Image classification models deployed in the real world may receive inputs outside the intended data distribution. For critical applications such as clinical decision making, it is important that a model can detect such out-of-distribution (OOD) inputs and express its uncertainty. In this work, we assess the capability of various state-of-the-art approaches for confidence-based OOD detection through a comparative study and in-depth analysis. First, we leverage a computer vision benchmark to reproduce and compare multiple OOD detection methods. We then evaluate their capabilities on the challenging task of disease classification using chest X-rays. Our study shows that high performance in a computer vision task does not directly translate to accuracy in a medical imaging task. We analyse factors that affect performance of the methods between the two tasks. Our results provide useful insights for developing the next generation of OOD detection methods.