MultiCaption: Detecting disinformation using multilingual visual claims
This addresses the scarcity of datasets for automated fact-checking in multimodal and multilingual environments, though it is incremental as it primarily provides a new dataset and baselines.
The authors tackled the problem of detecting disinformation in multilingual visual claims by introducing MultiCaption, a dataset of 11,088 visual claims in 64 languages, and found that it is more challenging than standard tasks, requiring task-specific finetuning for strong performance.
Online disinformation poses an escalating threat to society, driven increasingly by the rapid spread of misleading content across both multimedia and multilingual platforms. While automated fact-checking methods have advanced in recent years, their effectiveness remains constrained by the scarcity of datasets that reflect these real-world complexities. To address this gap, we first present MultiCaption, a new dataset specifically designed for detecting contradictions in visual claims. Pairs of claims referring to the same image or video were labeled through multiple strategies to determine whether they contradict each other. The resulting dataset comprises 11,088 visual claims in 64 languages, offering a unique resource for building and evaluating misinformation-detection systems in truly multimodal and multilingual environments. We then provide comprehensive experiments using transformer-based architectures, natural language inference models, and large language models, establishing strong baselines for future research. The results show that MultiCaption is more challenging than standard NLI tasks, requiring task-specific finetuning for strong performance. Moreover, the gains from multilingual training and testing highlight the dataset's potential for building effective multilingual fact-checking pipelines without relying on machine translation.