ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms
This work addresses the need for clearer evaluation in OOD detection for machine learning practitioners, though it is incremental as it focuses on dataset creation and analysis rather than proposing a new method.
The paper tackles the problem of evaluating out-of-distribution (OOD) detection algorithms by introducing ImageNet-OOD, a dataset that decouples semantic and covariate shifts, revealing that current detectors are more sensitive to covariate shift and show minimal benefits for semantic shift detection.
The task of out-of-distribution (OOD) detection is notoriously ill-defined. Earlier works focused on new-class detection, aiming to identify label-altering data distribution shifts, also known as "semantic shift." However, recent works argue for a focus on failure detection, expanding the OOD evaluation framework to account for label-preserving data distribution shifts, also known as "covariate shift." Intriguingly, under this new framework, complex OOD detectors that were previously considered state-of-the-art now perform similarly to, or even worse than the simple maximum softmax probability baseline. This raises the question: what are the latest OOD detectors actually detecting? Deciphering the behavior of OOD detection algorithms requires evaluation datasets that decouples semantic shift and covariate shift. To aid our investigations, we present ImageNet-OOD, a clean semantic shift dataset that minimizes the interference of covariate shift. Through comprehensive experiments, we show that OOD detectors are more sensitive to covariate shift than to semantic shift, and the benefits of recent OOD detection algorithms on semantic shift detection is minimal. Our dataset and analyses provide important insights for guiding the design of future OOD detectors.