When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Models
This work addresses the problem of ambiguity in visual evidence for vision models, which is significant for applications that rely on accurate image interpretation, such as self-driving cars or medical diagnosis.
The authors investigated how vision models interpret face-like patterns in ambiguous images, finding that different models exhibit distinct behaviors, such as semantic overactivation or uncertainty-as-abstention, with some models producing strong and confident over-calls, especially for negative emotions. The results show that behavior under ambiguity is governed more by representational choices than score thresholds, with low uncertainty sometimes signaling extreme over-interpretation.
When visual evidence is ambiguous, vision models must decide whether to interpret face-like patterns as meaningful. Face pareidolia, the perception of faces in non-face objects, provides a controlled probe of this behavior. We introduce a representation-level diagnostic framework that analyzes detection, localization, uncertainty, and bias across class, difficulty, and emotion in face pareidolia images. Under a unified protocol, we evaluate six models spanning four representational regimes: vision-language models (VLMs; CLIP-B/32, CLIP-L/14, LLaVA-1.5-7B), pure vision classification (ViT), general object detection (YOLOv8), and face detection (RetinaFace). Our analysis reveals three mechanisms of interpretation under ambiguity. VLMs exhibit semantic overactivation, systematically pulling ambiguous non-human regions toward the Human concept, with LLaVA-1.5-7B producing the strongest and most confident over-calls, especially for negative emotions. ViT instead follows an uncertainty-as-abstention strategy, remaining diffuse yet largely unbiased. Detection-based models achieve low bias through conservative priors that suppress pareidolia responses even when localization is controlled. These results show that behavior under ambiguity is governed more by representational choices than score thresholds, and that uncertainty and bias are decoupled: low uncertainty can signal either safe suppression, as in detectors, or extreme over-interpretation, as in VLMs. Pareidolia therefore provides a compact diagnostic and a source of ambiguity-aware hard negatives for probing and improving the semantic robustness of vision-language systems. Code will be released upon publication.