Vision Also You Need: Navigating Out-of-Distribution Detection with Multimodal Large Language Model
This work addresses OOD detection for multimodal AI systems, offering a novel approach that is incremental but improves performance in both near and far OOD tasks.
The paper tackles out-of-distribution (OOD) detection by proposing MM-OOD, a pipeline that uses multimodal large language models (MLLMs) to address over-reliance on text knowledge, achieving significant improvements on datasets like Food-101 and validating scalability on ImageNet-1K.
Out-of-Distribution (OOD) detection is a critical task that has garnered significant attention. The emergence of CLIP has spurred extensive research into zero-shot OOD detection, often employing a training-free approach. Current methods leverage expert knowledge from large language models (LLMs) to identify potential outliers. However, these approaches tend to over-rely on knowledge in the text space, neglecting the inherent challenges involved in detecting out-of-distribution samples in the image space. In this paper, we propose a novel pipeline, MM-OOD, which leverages the multimodal reasoning capabilities of MLLMs and their ability to conduct multi-round conversations for enhanced outlier detection. Our method is designed to improve performance in both near OOD and far OOD tasks. Specifically, (1) for near OOD tasks, we directly feed ID images and corresponding text prompts into MLLMs to identify potential outliers; and (2) for far OOD tasks, we introduce the sketch-generate-elaborate framework: first, we sketch outlier exposure using text prompts, then generate corresponding visual OOD samples, and finally elaborate by using multimodal prompts. Experiments demonstrate that our method achieves significant improvements on widely used multimodal datasets such as Food-101, while also validating its scalability on ImageNet-1K.