Sensitivity of Generative VLMs to Semantically and Lexically Altered Prompts
This work addresses a vulnerability in generative VLMs that affects their robustness and reliability for users relying on prompt-based interactions.
The paper investigates the sensitivity of generative vision-language models (VLMs) to lexical and semantic alterations in prompts, finding that they are highly sensitive to such changes, which negatively impacts output consistency techniques.
Despite the significant influx of prompt-tuning techniques for generative vision-language models (VLMs), it remains unclear how sensitive these models are to lexical and semantic alterations in prompts. In this paper, we evaluate the ability of generative VLMs to understand lexical and semantic changes in text using the SugarCrepe++ dataset. We analyze the sensitivity of VLMs to lexical alterations in prompts without corresponding semantic changes. Our findings demonstrate that generative VLMs are highly sensitive to such alterations. Additionally, we show that this vulnerability affects the performance of techniques aimed at achieving consistency in their outputs.