On the Domain Robustness of Contrastive Vision-Language Models
This work addresses the issue of domain robustness for practitioners relying on pretrained vision-language models, though it is incremental as it focuses on evaluation rather than improving the models themselves.
The authors tackled the problem of domain-specific robustness in vision-language models by introducing Deepbench, a framework that uses an LLM to generate realistic image corruptions for targeted evaluation, revealing substantial variability in robustness across six real-world domains.
In real-world vision-language applications, practitioners increasingly rely on large, pretrained foundation models rather than custom-built solutions, despite limited transparency regarding their training data and processes. While these models achieve impressive performance on general benchmarks, their effectiveness can decline notably under specialized domain shifts, such as unique imaging conditions or environmental variations. In this work, we introduce Deepbench, a framework designed to assess domain-specific robustness of vision-language models (VLMs). Deepbench leverages a large language model (LLM) to generate realistic, context-aware image corruptions tailored to specific deployment domains without requiring labeled data. We evaluate a range of contrastive vision-language architectures and architectural variants across six real-world domains and observe substantial variability in robustness, highlighting the need for targeted, domain-aware evaluation. Deepbench is released as open-source software to support further research into domain-aware robustness assessment.