Visual Interestingness Decoded: How GPT-4o Mirrors Human Interests
This work addresses the problem of understanding and predicting visual interestingness for applications in attention-based systems, but it is incremental as it builds on existing LMM capabilities.
The paper investigates how well GPT-4o captures visual interestingness by comparing its predictions to human assessments, finding partial alignment and using this to train a learning-to-rank model for labeling image pairs.
Our daily life is highly influenced by what we consume and see. Attracting and holding one's attention -- the definition of (visual) interestingness -- is essential. The rise of Large Multimodal Models (LMMs) trained on large-scale visual and textual data has demonstrated impressive capabilities. We explore these models' potential to understand to what extent the concepts of visual interestingness are captured and examine the alignment between human assessments and GPT-4o's, a leading LMM, predictions through comparative analysis. Our studies reveal partial alignment between humans and GPT-4o. It already captures the concept as best compared to state-of-the-art methods. Hence, this allows for the effective labeling of image pairs according to their (commonly) interestingness, which are used as training data to distill the knowledge into a learning-to-rank model. The insights pave the way for a deeper understanding of human interest.