Can Argus Judge Them All? Comparing VLMs Across Domains
This work addresses the problem of inconsistent performance across domains for vision-language models, providing insights for industrial deployment and guiding development toward more robust architectures.
The paper benchmarks CLIP, BLIP, and LXMERT across diverse vision-language tasks, finding that CLIP generalizes best with a Cross-Dataset Consistency score of 0.92, while BLIP excels on curated data and LXMERT leads in structured reasoning.
Vision-Language Models (VLMs) are advancing multimodal AI, yet their performance consistency across tasks is underexamined. We benchmark CLIP, BLIP, and LXMERT across diverse datasets spanning retrieval, captioning, and reasoning. Our evaluation includes task accuracy, generation quality, efficiency, and a novel Cross-Dataset Consistency (CDC) metric. CLIP shows strongest generalization (CDC: 0.92), BLIP excels on curated data, and LXMERT leads in structured reasoning. These results expose trade-offs between generalization and specialization, informing industrial deployment of VLMs and guiding development toward robust, task-flexible architectures.