Toward a Realistic Benchmark for Out-of-Distribution Detection
This work addresses the need for more realistic benchmarks in OOD detection, which is crucial for deploying reliable deep neural networks in safety-critical applications, though it is incremental as it builds on existing datasets and methods.
The paper tackles the problem of evaluating out-of-distribution (OOD) detection techniques by highlighting that existing benchmarks often use far-OOD samples, which fail to capture real-world complexity. It introduces a new benchmark based on ImageNet and Places365 that uses semantic similarity to define in-distribution and OOD classes, showing that measured efficacy varies with the benchmark and confidence-based techniques can outperform classifier-based ones on near-OOD samples.
Deep neural networks are increasingly used in a wide range of technologies and services, but remain highly susceptible to out-of-distribution (OOD) samples, that is, drawn from a different distribution than the original training set. A common approach to address this issue is to endow deep neural networks with the ability to detect OOD samples. Several benchmarks have been proposed to design and validate OOD detection techniques. However, many of them are based on far-OOD samples drawn from very different distributions, and thus lack the complexity needed to capture the nuances of real-world scenarios. In this work, we introduce a comprehensive benchmark for OOD detection, based on ImageNet and Places365, that assigns individual classes as in-distribution or out-of-distribution depending on the semantic similarity with the training set. Several techniques can be used to determine which classes should be considered in-distribution, yielding benchmarks with varying properties. Experimental results on different OOD detection techniques show how their measured efficacy depends on the selected benchmark and how confidence-based techniques may outperform classifier-based ones on near-OOD samples.