Hummus: A Dataset of Humorous Multimodal Metaphor Use
This work addresses a gap in computational humor and metaphor research for AI and linguistics communities, but it is incremental as it builds on existing theories and datasets.
The authors tackled the understudied problem of humorous multimodal metaphors by creating the Hummus Dataset with expert annotations on 1,000 image-caption pairs, and found that current multimodal large language models struggle to detect and understand these metaphors, especially in integrating visual and textual information.
Metaphor and humor share a lot of common ground, and metaphor is one of the most common humorous mechanisms. This study focuses on the humorous capacity of multimodal metaphors, which has not received due attention in the community. We take inspiration from the Incongruity Theory of humor, the Conceptual Metaphor Theory, and the annotation scheme behind the VU Amsterdam Metaphor Corpus, and developed a novel annotation scheme for humorous multimodal metaphor use in image-caption pairs. We create the Hummus Dataset of Humorous Multimodal Metaphor Use, providing expert annotation on 1k image-caption pairs sampled from the New Yorker Caption Contest corpus. Using the dataset, we test state-of-the-art multimodal large language models (MLLMs) on their ability to detect and understand humorous multimodal metaphor use. Our experiments show that current MLLMs still struggle with processing humorous multimodal metaphors, particularly with regard to integrating visual and textual information. We release our dataset and code at github.com/xiaoyuisrain/humorous-multimodal-metaphor-use.