Large Language Models and Provenance Metadata for Determining the Relevance of Images and Videos in News Stories
This addresses the challenge of detecting misinformation in multimodal content for news verification, though it appears incremental as it builds on existing LLM and metadata techniques.
The paper tackles the problem of multimodal misinformation by analyzing text and provenance metadata of images and videos in news stories to determine relevance, resulting in an open-sourced system prototype and interactive web interface.
The most effective misinformation campaigns are multimodal, often combining text with images and videos taken out of context -- or fabricating them entirely -- to support a given narrative. Contemporary methods for detecting misinformation, whether in deepfakes or text articles, often miss the interplay between multiple modalities. Built around a large language model, the system proposed in this paper addresses these challenges. It analyzes both the article's text and the provenance metadata of included images and videos to determine whether they are relevant. We open-source the system prototype and interactive web interface.