The Effects of Visual Priming on Cooperative Behavior in Vision-Language Models
For researchers and developers deploying VLMs in visually rich, safety-critical environments, this work highlights the need for robust evaluation frameworks to account for visual input biases.
This paper shows that visual priming (images depicting behavioral concepts and color-coded reward matrices) can influence the cooperative behavior of Vision-Language Models in the Iterated Prisoner's Dilemma, with varying susceptibility across models. Mitigation strategies like prompt modifications and Chain of Thought reasoning had mixed effectiveness.
As Vision-Language Models (VLMs) become increasingly integrated into decision-making systems, it is essential to understand how visual inputs influence their behavior. This paper investigates the effects of visual priming on VLMs' cooperative behavior using the Iterated Prisoner's Dilemma (IPD) as a test scenario. We examine whether exposure to images depicting behavioral concepts (kindness/helpfulness vs. aggressiveness/selfishness) and color-coded reward matrices alters VLM decision patterns. Experiments were conducted across multiple state-of-the-art VLMs. We further explore mitigation strategies including prompt modifications, Chain of Thought (CoT) reasoning, and visual token reduction. Results show that VLM behavior can be influenced by both image content and color cues, with varying susceptibility and mitigation effectiveness across models. These findings not only underscore the importance of robust evaluation frameworks for VLM deployment in visually rich and safety-critical environments, but also highlight how architectural and training differences among models may lead to distinct behavioral responses-an area worthy of further investigation.