Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations
This highlights a robustness problem for stakeholders relying on these models in applications requiring trustworthy decision-making, though it is incremental as it focuses on a specific vulnerability rather than a broader breakthrough.
The paper identifies a vulnerability in large language and vision-language models where they are sensitive to adversarial permutations in answer sets for multiple-choice question answering, showing that this issue persists across various model sizes and recent architectures.
Large language and vision-language models are rapidly being deployed in practice thanks to their impressive capabilities in instruction following, in-context learning, and so on. This raises an urgent need to carefully analyse their robustness so that stakeholders can understand if and when such models are trustworthy enough to be relied upon in any given application. In this paper, we highlight a specific vulnerability in popular models, namely permutation sensitivity in multiple-choice question answering (MCQA). Specifically, we show empirically that popular models are vulnerable to adversarial permutation in answer sets for multiple-choice prompting, which is surprising as models should ideally be as invariant to prompt permutation as humans are. These vulnerabilities persist across various model sizes, and exist in very recent language and vision-language models. Code is available at https://github.com/ys-zong/FoolyourVLLMs.