Respecting Modality Gap in Post-hoc Out-of-distribution Detection with Pre-trained Vision-Language Models
For practitioners using pre-trained VLMs for zero-shot OOD detection, this work addresses a fundamental limitation of the text-as-prototype paradigm with a practical post-hoc solution.
This paper identifies a modality gap between textual and visual prototypes in vision-language models for OOD detection, and proposes an online pseudo-supervised framework to learn visual prototypes from test-time data, achieving state-of-the-art performance across multiple benchmarks.
Out-of-distribution (OOD) detection has emerged as a popular technique to enhance the reliability of machine learning models by identifying unexpected inputs from unknown classes. Recent progress in pre-trained vision-language models (VLMs) has enabled zero-shot OOD detection without access to in-distribution (ID) training data; in this setting, existing methods commonly treat text embeddings of class names as class prototypes. In this paper, we challenge the widely adopted text-as-prototype paradigm by theoretically showing that off-the-shelf textual prototypes are generally misaligned with the optimal visual prototypes, yielding an intrinsic modality gap that cannot be eliminated by prompt engineering alone. To mitigate this gap under the post-hoc constraint, this paper presents an online pseudo-supervised framework that directly learns class prototypes in the visual feature space using unlabeled test-time data streams and soft predictions from the pre-trained VLMs. We provide theoretical guarantees for the convergence of the online optimization procedure. Extensive experiments empirically demonstrate that our method achieves a new state of the art across a variety of OOD detection setups.