MetaSumPerceiver: Multimodal Multi-Document Evidence Summarization for Fact-Checking
This addresses the laborious task of fact-checking real-world claims by automating evidence summarization from multiple sources, though it is incremental as it builds on existing summarization and multimodal methods.
The paper tackles the problem of summarizing multimodal, multi-document evidence for fact-checking by introducing a model that generates claim-specific summaries, outperforming the state-of-the-art by 4.6% on the MOCHEG dataset and showing strong performance on a new dataset.
Fact-checking real-world claims often requires reviewing multiple multimodal documents to assess a claim's truthfulness, which is a highly laborious and time-consuming task. In this paper, we present a summarization model designed to generate claim-specific summaries useful for fact-checking from multimodal, multi-document datasets. The model takes inputs in the form of documents, images, and a claim, with the objective of assisting in fact-checking tasks. We introduce a dynamic perceiver-based model that can handle inputs from multiple modalities of arbitrary lengths. To train our model, we leverage a novel reinforcement learning-based entailment objective to generate summaries that provide evidence distinguishing between different truthfulness labels. To assess the efficacy of our approach, we conduct experiments on both an existing benchmark and a new dataset of multi-document claims that we contribute. Our approach outperforms the SOTA approach by 4.6% in the claim verification task on the MOCHEG dataset and demonstrates strong performance on our new Multi-News-Fact-Checking dataset.