Reject option models comprising out-of-distribution detection
This addresses the fundamental challenge of handling OOD data in machine learning, offering improved methods for reliable model deployment, though it is incremental as it builds on existing reject option frameworks.
The paper tackles the problem of optimal prediction strategies for out-of-distribution (OOD) setups by proposing three reject option models and showing they share a common optimal strategy, leading to double-score OOD methods that outperform state-of-the-art approaches in experiments.
The optimal prediction strategy for out-of-distribution (OOD) setups is a fundamental question in machine learning. In this paper, we address this question and present several contributions. We propose three reject option models for OOD setups: the Cost-based model, the Bounded TPR-FPR model, and the Bounded Precision-Recall model. These models extend the standard reject option models used in non-OOD setups and define the notion of an optimal OOD selective classifier. We establish that all the proposed models, despite their different formulations, share a common class of optimal strategies. Motivated by the optimal strategy, we introduce double-score OOD methods that leverage uncertainty scores from two chosen OOD detectors: one focused on OOD/ID discrimination and the other on misclassification detection. The experimental results consistently demonstrate the superior performance of this simple strategy compared to state-of-the-art methods. Additionally, we propose novel evaluation metrics derived from the definition of the optimal strategy under the proposed OOD rejection models. These new metrics provide a comprehensive and reliable assessment of OOD methods without the deficiencies observed in existing evaluation approaches.