Video Manipulations Beyond Faces: A Dataset with Human-Machine Analysis
This addresses the spread of misinformation online by providing a more comprehensive dataset for video manipulation detection, though it is incremental as it builds on existing deepfake datasets.
The authors tackled the problem of detecting manipulated video content beyond facial alterations by introducing VideoSham, a dataset of 826 videos with diverse spatial and temporal attacks, and found that state-of-the-art detection algorithms perform poorly on it, with a user study involving 1200 participants revealing gaps in human and AI capabilities.
As tools for content editing mature, and artificial intelligence (AI) based algorithms for synthesizing media grow, the presence of manipulated content across online media is increasing. This phenomenon causes the spread of misinformation, creating a greater need to distinguish between ``real'' and ``manipulated'' content. To this end, we present VideoSham, a dataset consisting of 826 videos (413 real and 413 manipulated). Many of the existing deepfake datasets focus exclusively on two types of facial manipulations -- swapping with a different subject's face or altering the existing face. VideoSham, on the other hand, contains more diverse, context-rich, and human-centric, high-resolution videos manipulated using a combination of 6 different spatial and temporal attacks. Our analysis shows that state-of-the-art manipulation detection algorithms only work for a few specific attacks and do not scale well on VideoSham. We performed a user study on Amazon Mechanical Turk with 1200 participants to understand if they can differentiate between the real and manipulated videos in VideoSham. Finally, we dig deeper into the strengths and weaknesses of performances by humans and SOTA-algorithms to identify gaps that need to be filled with better AI algorithms. We present the dataset at https://github.com/adobe-research/VideoSham-dataset.