An Empirical Analysis of VLM-based OOD Detection: Mechanisms, Advantages, and Sensitivity
It addresses the need for reliable AI systems by providing empirical insights into VLM-based OOD detection, though it is incremental as it builds on existing capabilities.
This paper tackles the problem of understanding why Vision-Language Models (VLMs) like CLIP are effective for zero-shot out-of-distribution (OOD) detection by analyzing their mechanisms, advantages over single-modal methods, and sensitivity to prompts, finding they leverage semantic novelty but are highly sensitive to prompt phrasing.
Vision-Language Models (VLMs), such as CLIP, have demonstrated remarkable zero-shot out-of-distribution (OOD) detection capabilities, vital for reliable AI systems. Despite this promising capability, a comprehensive understanding of (1) why they work so effectively, (2) what advantages do they have over single-modal methods, and (3) how is their behavioral robustness -- remains notably incomplete within the research community. This paper presents a systematic empirical analysis of VLM-based OOD detection using in-distribution (ID) and OOD prompts. (1) Mechanisms: We systematically characterize and formalize key operational properties within the VLM embedding space that facilitate zero-shot OOD detection. (2) Advantages: We empirically quantify the superiority of these models over established single-modal approaches, attributing this distinct advantage to the VLM's capacity to leverage rich semantic novelty. (3) Sensitivity: We uncovers a significant and previously under-explored asymmetry in their robustness profile: while exhibiting resilience to common image noise, these VLM-based methods are highly sensitive to prompt phrasing. Our findings contribute a more structured understanding of the strengths and critical vulnerabilities inherent in VLM-based OOD detection, offering crucial, empirically-grounded guidance for developing more robust and reliable future designs.