Evaluating Robustness of Vision-Language Models Under Noisy Conditions
This work addresses robustness issues in VLMs for multimodal applications, providing a standardized benchmark, but it is incremental as it focuses on evaluation rather than new methods.
The study evaluated the robustness of Vision-Language Models under noisy conditions like lighting variation and motion blur, finding that certain noise types dramatically degrade performance across models, with larger models not always outperforming smaller ones.
Vision-Language Models (VLMs) have attained exceptional success across multimodal tasks such as image captioning and visual question answering. However, their robustness under noisy conditions remains unfamiliar. In this study, we present a comprehensive evaluation framework to evaluate the performance of several state-of-the-art VLMs under controlled perturbations, including lighting variation, motion blur, and compression artifacts. We used both lexical-based metrics (BLEU, METEOR, ROUGE, CIDEr) and neural-based similarity measures using sentence embeddings to quantify semantic alignment. Our experiments span diverse datasets, revealing key insights: (1) descriptiveness of ground-truth captions significantly influences model performance; (2) larger models like LLaVA excel in semantic understanding but do not universally outperform smaller models; and (3) certain noise types, such as JPEG compression and motion blur, dramatically degrade performance across models. Our findings highlight the nuanced trade-offs between model size, dataset characteristics, and noise resilience, offering a standardized benchmark for future robust multimodal learning.