CITE: A Corpus of Image-Text Discourse Relations
This provides a dataset for researchers in multimodal AI to study image-text interactions, though it is incremental as it builds on existing text-based discourse corpora.
The authors tackled the problem of understanding multimodal discourse by creating a crowd-sourced corpus of image-text coherence relations in cooking recipes, resulting in a new resource that aids in analyzing natural communication and common-sense reasoning.
This paper presents a novel crowd-sourced resource for multimodal discourse: our resource characterizes inferences in image-text contexts in the domain of cooking recipes in the form of coherence relations. Like previous corpora annotating discourse structure between text arguments, such as the Penn Discourse Treebank, our new corpus aids in establishing a better understanding of natural communication and common-sense reasoning, while our findings have implications for a wide range of applications, such as understanding and generation of multimodal documents.