On the Limitations of Vision-Language Models in Understanding Image Transforms
This work addresses a critical bottleneck in VLMs for applications like image editing, but it is incremental as it builds on existing models and datasets.
The paper investigates the limitations of Vision-Language Models (VLMs) like CLIP and SigLIP in understanding basic image transformations, revealing that they struggle with multiple augmentations, and evaluates how this affects downstream tasks such as image editing.
Vision Language Models (VLMs) have demonstrated significant potential in various downstream tasks, including Image/Video Generation, Visual Question Answering, Multimodal Chatbots, and Video Understanding. However, these models often struggle with basic image transformations. This paper investigates the image-level understanding of VLMs, specifically CLIP by OpenAI and SigLIP by Google. Our findings reveal that these models lack comprehension of multiple image-level augmentations. To facilitate this study, we created an augmented version of the Flickr8k dataset, pairing each image with a detailed description of the applied transformation. We further explore how this deficiency impacts downstream tasks, particularly in image editing, and evaluate the performance of state-of-the-art Image2Image models on simple transformations.