Analyzing CLIP's Performance Limitations in Multi-Object Scenarios: A Controlled High-Resolution Study
This work identifies specific biases in CLIP that affect its reliability in complex visual tasks, providing insights for improving vision-language models, though it is incremental as it focuses on analyzing existing limitations rather than proposing new solutions.
This study analyzed CLIP's performance limitations in multi-object scenarios, revealing significant biases in its image and text encoders that favor larger objects and first-mentioned objects, respectively, and showing these biases impact tasks like image-caption matching and text-to-image generation.
Contrastive Language-Image Pre-training (CLIP) models have demonstrated remarkable performance in zero-shot classification tasks, yet their efficacy in handling complex multi-object scenarios remains challenging. This study presents a comprehensive analysis of CLIP's performance limitations in multi-object contexts through controlled experiments. We introduce two custom datasets, SimCO and CompCO, to evaluate CLIP's image and text encoders in various multi-object configurations. Our findings reveal significant biases in both encoders: the image encoder favors larger objects, while the text encoder prioritizes objects mentioned first in descriptions. We hypothesize these biases originate from CLIP's training process and provide evidence through analyses of the COCO dataset and CLIP's training progression. Additionally, we extend our investigation to Stable Diffusion models, revealing that biases in the CLIP text encoder significantly impact text-to-image generation tasks. Our experiments demonstrate how these biases affect CLIP's performance in image-caption matching and generation tasks, particularly when manipulating object sizes and their order in captions. This work contributes valuable insights into CLIP's behavior in complex visual environments and highlights areas for improvement in future vision-language models.