When Relations Break: Analyzing Relation Hallucination in Vision-Language Model Under Rotation and Noise
For researchers developing robust vision-language models, this work identifies a critical gap between perceptual robustness and relational understanding.
The paper shows that mild visual perturbations like rotation and noise significantly degrade relational reasoning in vision-language models, and existing augmentation and preprocessing strategies only partially mitigate this issue.
Vision-language models (VLMs) achieve strong multimodal performance but remain prone to relation hallucination, which requires accurate reasoning over inter-object interactions. We study the impact of visual perturbations, specifically rotation and noise, and show that even mild distortions significantly degrade relational reasoning across models and datasets. We further evaluate prompt-based augmentation and preprocessing strategies (orientation correction and denoising), finding that while they offer partial improvements, they do not fully resolve hallucinations. Our results reveal a gap between perceptual robustness and relational understanding, highlighting the need for more robust, geometry-aware VLMs.