Challenges in Understanding Modality Conflict in Vision-Language Models
This provides insights for improving model robustness in multimodal AI applications, though it is incremental research on existing models.
This paper investigates how Vision-Language Models handle conflicting multimodal inputs, finding that conflict detection and resolution are distinct mechanisms with linearly decodable signals in intermediate layers and divergent attention patterns.
This paper highlights the challenge of decomposing conflict detection from conflict resolution in Vision-Language Models (VLMs) and presents potential approaches, including using a supervised metric via linear probes and group-based attention pattern analysis. We conduct a mechanistic investigation of LLaVA-OV-7B, a state-of-the-art VLM that exhibits diverse resolution behaviors when faced with conflicting multimodal inputs. Our results show that a linearly decodable conflict signal emerges in the model's intermediate layers and that attention patterns associated with conflict detection and resolution diverge at different stages of the network. These findings support the hypothesis that detection and resolution are functionally distinct mechanisms. We discuss how such decomposition enables more actionable interpretability and targeted interventions for improving model robustness in challenging multimodal settings.