Practical Evaluation of Out-of-Distribution Detection Methods for Image Classification
This work addresses the problem of generalizability and real-world applicability of OOD detection method evaluations for practitioners, providing guidance for method selection.
This paper re-evaluates out-of-distribution (OOD) detection methods for image recognition across three distinct scenarios: irrelevant input detection, novel class detection, and domain shift detection. The study found that the relative performance of different OOD detection methods varies significantly depending on the specific scenario and dataset used.
We reconsider the evaluation of OOD detection methods for image recognition. Although many studies have been conducted so far to build better OOD detection methods, most of them follow Hendrycks and Gimpel's work for the method of experimental evaluation. While the unified evaluation method is necessary for a fair comparison, there is a question of if its choice of tasks and datasets reflect real-world applications and if the evaluation results can generalize to other OOD detection application scenarios. In this paper, we experimentally evaluate the performance of representative OOD detection methods for three scenarios, i.e., irrelevant input detection, novel class detection, and domain shift detection, on various datasets and classification tasks. The results show that differences in scenarios and datasets alter the relative performance among the methods. Our results can also be used as a guide for practitioners for the selection of OOD detection methods.