Deep Multimodal Image-Repurposing Detection
This addresses the challenge of combating misinformation on social media by detecting image repurposing, though it is incremental as it builds on existing multimodal detection approaches.
The paper tackles the problem of detecting image repurposing, where unmanipulated images are paired with incorrect metadata to spread misinformation, by introducing the MEIR dataset and a deep multimodal learning model. The model outperforms state-of-the-art methods with AUC improvements up to 0.23 on existing and new datasets.
Nefarious actors on social media and other platforms often spread rumors and falsehoods through images whose metadata (e.g., captions) have been modified to provide visual substantiation of the rumor/falsehood. This type of modification is referred to as image repurposing, in which often an unmanipulated image is published along with incorrect or manipulated metadata to serve the actor's ulterior motives. We present the Multimodal Entity Image Repurposing (MEIR) dataset, a substantially challenging dataset over that which has been previously available to support research into image repurposing detection. The new dataset includes location, person, and organization manipulations on real-world data sourced from Flickr. We also present a novel, end-to-end, deep multimodal learning model for assessing the integrity of an image by combining information extracted from the image with related information from a knowledge base. The proposed method is compared against state-of-the-art techniques on existing datasets as well as MEIR, where it outperforms existing methods across the board, with AUC improvement up to 0.23.