MEG: Multi-Evidence GNN for Multimodal Semantic Forensics
This work provides a method for improving the detection of fake news involving semantic manipulations across modalities, which is a problem for social media platforms and users.
This paper addresses multimodal semantic forensics, specifically image repurposing, where an image is semantically misrepresented by its accompanying metadata. The authors propose a graph neural network model that utilizes multiple retrieved multimedia packages as evidence, achieving an error reduction of up to 25% compared to state-of-the-art algorithms.
Fake news often involves semantic manipulations across modalities such as image, text, location etc and requires the development of multimodal semantic forensics for its detection. Recent research has centered the problem around images, calling it image repurposing -- where a digitally unmanipulated image is semantically misrepresented by means of its accompanying multimodal metadata such as captions, location, etc. The image and metadata together comprise a multimedia package. The problem setup requires algorithms to perform multimodal semantic forensics to authenticate a query multimedia package using a reference dataset of potentially related packages as evidences. Existing methods are limited to using a single evidence (retrieved package), which ignores potential performance improvement from the use of multiple evidences. In this work, we introduce a novel graph neural network based model for multimodal semantic forensics, which effectively utilizes multiple retrieved packages as evidences and is scalable with the number of evidences. We compare the scalability and performance of our model against existing methods. Experimental results show that the proposed model outperforms existing state-of-the-art algorithms with an error reduction of up to 25%.