Unveiling the Invisible: Captioning Videos with Metaphors
This work addresses a gap in vision-language understanding for complex language phenomena like metaphors in videos, which could benefit fields such as creative AI and multimedia analysis, though it is incremental as it builds on existing video language models.
The paper tackles the problem of generating captions for metaphors in videos, a previously unexplored task, by introducing a manually created dataset of 705 videos with 2115 captions and a new metric called Average Concept Distance (ACD) for evaluation. It proposes a low-resource system, GIT-LLaVA, which achieves comparable performance to state-of-the-art video language models on this task.
Metaphors are a common communication tool used in our day-to-day life. The detection and generation of metaphors in textual form have been studied extensively but metaphors in other forms have been under-explored. Recent studies have shown that Vision-Language (VL) models cannot understand visual metaphors in memes and adverts. As of now, no probing studies have been done that involve complex language phenomena like metaphors with videos. Hence, we introduce a new VL task of describing the metaphors present in the videos in our work. To facilitate this novel task, we construct and release a manually created dataset with 705 videos and 2115 human-written captions, along with a new metric called Average Concept Distance (ACD), to automatically evaluate the creativity of the metaphors generated. We also propose a novel low-resource video metaphor captioning system: GIT-LLaVA, which obtains comparable performance to SoTA video language models on the proposed task. We perform a comprehensive analysis of existing video language models on this task and publish our dataset, models, and benchmark results to enable further research.