A Framework for Generating Semantically Ambiguous Images to Probe Human and Machine Perception
This work addresses the challenge of bridging human psychophysics and machine vision interpretability for researchers in AI and cognitive science, though it is incremental as it builds on existing CLIP and generative models.
The authors tackled the problem of comparing human and machine perception of ambiguous images by developing a framework that generates continuous spectra of semantically ambiguous images using CLIP embeddings, and found that machine classifiers are more biased towards seeing 'rabbit' while humans align more closely with the CLIP embeddings, with the guidance scale affecting human sensitivity more strongly.
The classic duck-rabbit illusion reveals that when visual evidence is ambiguous, the human brain must decide what it sees. But where exactly do human observers draw the line between ''duck'' and ''rabbit'', and do machine classifiers draw it in the same place? We use semantically ambiguous images as interpretability probes to expose how vision models represent the boundaries between concepts. We present a psychophysically-informed framework that interpolates between concepts in the CLIP embedding space to generate continuous spectra of ambiguous images, allowing us to precisely measure where and how humans and machine classifiers place their semantic boundaries. Using this framework, we show that machine classifiers are more biased towards seeing ''rabbit'', whereas humans are more aligned with the CLIP embedding used for synthesis, and the guidance scale seems to affect human sensitivity more strongly than machine classifiers. Our framework demonstrates how controlled ambiguity can serve as a diagnostic tool to bridge the gap between human psychophysical analysis, image classification, and generative image models, offering insight into human-model alignment, robustness, model interpretability, and image synthesis methods.