Mining Contextualized Visual Associations from Images for Creativity Understanding
This work addresses the challenge of understanding creative output in vision-language models, which is incremental as it builds on existing methods like CLIP by enhancing caption generation with mined associations.
The paper tackles the problem of generating creative captions for images by mining contextualized visual associations from unlabeled datasets, resulting in a new dataset of 1.7 million creative captions for MSCOCO images and improvements in zero-shot image-text retrieval for creative domains like poetry and metaphor visualization.
Understanding another person's creative output requires a shared language of association. However, when training vision-language models such as CLIP, we rely on web-scraped datasets containing short, predominantly literal, alt-text. In this work, we introduce a method for mining contextualized associations for salient visual elements in an image that can scale to any unlabeled dataset. Given an image, we can use these mined associations to generate high quality creative captions at increasing degrees of abstraction. With our method, we produce a new dataset of visual associations and 1.7m creative captions for the images in MSCOCO. Human evaluation confirms that these captions remain visually grounded while exhibiting recognizably increasing abstraction. Moreover, fine-tuning a visual encoder on this dataset yields meaningful improvements in zero-shot image-text retrieval in two creative domains: poetry and metaphor visualization. We release our dataset, our generation code and our models for use by the broader community.