On the Brittleness of CLIP Text Encoders
This work addresses robustness issues in vision-language models for users relying on zero-shot classification and retrieval, highlighting a critical evaluation dimension beyond accuracy, though it is incremental as it focuses on analysis rather than new solutions.
The paper systematically analyzes the brittleness of CLIP text encoders to non-semantic query perturbations in multimedia information retrieval, finding that syntactic and semantic changes cause the largest instabilities, with trivial edits like punctuation and case also contributing significantly.
Multimodal co-embedding models, especially CLIP, have advanced the state of the art in zero-shot classification and multimedia information retrieval in recent years by aligning images and text in a shared representation space. However, such modals trained on a contrastive alignment can lack stability towards small input perturbations. Especially when dealing with manually expressed queries, minor variations in the query can cause large differences in the ranking of the best-matching results. In this paper, we present a systematic analysis of the effect of multiple classes of non-semantic query perturbations in an multimedia information retrieval scenario. We evaluate a diverse set of lexical, syntactic, and semantic perturbations across multiple CLIP variants using the TRECVID Ad-Hoc Video Search queries and the V3C1 video collection. Across models, we find that syntactic and semantic perturbations drive the largest instabilities, while brittleness is concentrated in trivial surface edits such as punctuation and case. Our results highlight robustness as a critical dimension for evaluating vision-language models beyond benchmark accuracy.