Delving into Out-of-Distribution Detection with Vision-Language Representations
This addresses the problem of reliable OOD detection for machine learning systems in open-world settings, offering a novel multi-modal approach that is incremental over existing single-modal methods.
The paper tackles out-of-distribution (OOD) detection by shifting from single-modal to multi-modal methods, proposing Maximum Concept Matching (MCM) which uses vision-language representations to achieve a 13.1% AUROC improvement over visual-only baselines on a hard OOD task.
Recognizing out-of-distribution (OOD) samples is critical for machine learning systems deployed in the open world. The vast majority of OOD detection methods are driven by a single modality (e.g., either vision or language), leaving the rich information in multi-modal representations untapped. Inspired by the recent success of vision-language pre-training, this paper enriches the landscape of OOD detection from a single-modal to a multi-modal regime. Particularly, we propose Maximum Concept Matching (MCM), a simple yet effective zero-shot OOD detection method based on aligning visual features with textual concepts. We contribute in-depth analysis and theoretical insights to understand the effectiveness of MCM. Extensive experiments demonstrate that MCM achieves superior performance on a wide variety of real-world tasks. MCM with vision-language features outperforms a common baseline with pure visual features on a hard OOD task with semantically similar classes by 13.1% (AUROC). Code is available at https://github.com/deeplearning-wisc/MCM.