DOI: Divergence-based Out-of-Distribution Indicators via Deep Generative Models
This addresses the need for reliable OoD detection in machine learning, offering a novel criterion that could advance the field beyond traditional likelihood-based methods.
The paper tackles the problem of evaluating out-of-distribution (OoD) indicators by conducting a large benchmark with 92 dataset pairs, revealing that existing methods perform poorly, and proposes a divergence-based framework and Single-shot Fine-tune algorithm that improves AUROC by 5-8 points.
To ensure robust and reliable classification results, OoD (out-of-distribution) indicators based on deep generative models are proposed recently and are shown to work well on small datasets. In this paper, we conduct the first large collection of benchmarks (containing 92 dataset pairs, which is 1 order of magnitude larger than previous ones) for existing OoD indicators and observe that none perform well. We thus advocate that a large collection of benchmarks is mandatory for evaluating OoD indicators. We propose a novel theoretical framework, DOI, for divergence-based Out-of-Distribution indicators (instead of traditional likelihood-based) in deep generative models. Following this framework, we further propose a simple and effective OoD detection algorithm: Single-shot Fine-tune. It significantly outperforms past works by 5~8 in AUROC, and its performance is close to optimal. In recent, the likelihood criterion is shown to be ineffective in detecting OoD. Single-shot Fine-tune proposes a novel fine-tune criterion to detect OoD, by whether the likelihood of the testing sample is improved after fine-tuning a well-trained model on it. Fine-tune criterion is a clear and easy-following criterion, which will lead the OoD domain into a new stage.