Diagnosing Vision Language Models' Perception by Leveraging Human Methods for Color Vision Deficiencies
This addresses the problem of perceptual inclusiveness and fairness in multimodal AI for diverse users, but it is incremental as it focuses on a specific aspect of perception.
The study evaluated Vision Language Models' ability to handle individual perceptual variation in color vision using the Ishihara Test, finding that while they can explain Color Vision Deficiencies in language, they fail to simulate perception in image-based tasks.
Large-scale Vision Language Models (LVLMs) are increasingly being applied to a wide range of real-world multimodal applications, involving complex visual and linguistic reasoning. As these models become more integrated into practical use, they are expected to handle complex aspects of human interaction. Among these, color perception is a fundamental yet highly variable aspect of visual understanding. It differs across individuals due to biological factors such as Color Vision Deficiencies (CVDs), as well as differences in culture and language. Despite its importance, perceptual diversity has received limited attention. In our study, we evaluate LVLMs' ability to account for individual level perceptual variation using the Ishihara Test, a widely used method for detecting CVDs. Our results show that LVLMs can explain CVDs in natural language, but they cannot simulate how people with CVDs perceive color in image based tasks. These findings highlight the need for multimodal systems that can account for color perceptual diversity and support broader discussions on perceptual inclusiveness and fairness in multimodal AI.