Caption-Driven Explorations: Aligning Image and Text Embeddings through Human-Inspired Foveated Vision
This work addresses the challenge of task-driven image exploration for vision science and AI researchers, though it appears incremental as it builds on existing CLIP and NeVA methods.
The researchers tackled the problem of predicting human visual attention during captioning tasks by introducing a new dataset (CapMIT1003) and a zero-shot method (NevaClip) that combines CLIP models with NeVA algorithms to generate scanpaths. The result showed that simulated scanpaths outperformed existing human attention models in plausibility for both captioning and free-viewing tasks.
Understanding human attention is crucial for vision science and AI. While many models exist for free-viewing, less is known about task-driven image exploration. To address this, we introduce CapMIT1003, a dataset with captions and click-contingent image explorations, to study human attention during the captioning task. We also present NevaClip, a zero-shot method for predicting visual scanpaths by combining CLIP models with NeVA algorithms. NevaClip generates fixations to align the representations of foveated visual stimuli and captions. The simulated scanpaths outperform existing human attention models in plausibility for captioning and free-viewing tasks. This research enhances the understanding of human attention and advances scanpath prediction models.