Contrast Sensitivity in Multimodal Large Language Models: A Psychophysics-Inspired Evaluation
This work addresses the need for scalable diagnostic tools to assess perceptual abilities in MLLMs, which is incremental as it adapts existing psychophysical methods to a new domain.
The study tackled the problem of evaluating how Multimodal Large Language Models (MLLMs) process low-level visual features by introducing a psychophysics-inspired method to estimate Contrast Sensitivity Functions (CSFs), revealing that some models resemble human CSFs in shape or scale but none capture both, and showing that CSFs predict model performance under frequency-filtered and adversarial conditions.
Understanding how Multimodal Large Language Models (MLLMs) process low-level visual features is critical for evaluating their perceptual abilities and has not been systematically characterized. Inspired by human psychophysics, we introduce a behavioural method for estimating the Contrast Sensitivity Function (CSF) in MLLMs by treating them as end-to-end observers. Models are queried with structured prompts while viewing noise-based stimuli filtered at specific spatial frequencies. Psychometric functions are derived from the binary verbal responses, and contrast thresholds (and CSFs) are obtained without relying on internal activations or classifier-based proxies. Our results reveal that some models resemble human CSFs in shape or scale, but none capture both. We also find that CSF estimates are highly sensitive to prompt phrasing, indicating limited linguistic robustness. Finally, we show that CSFs predict model performance under frequency-filtered and adversarial conditions. These findings highlight systematic differences in frequency tuning across MLLMs and establish CSF estimation as a scalable diagnostic tool for multimodal perception.