Deepfake Caricatures: Amplifying attention to artifacts increases deepfake detection by humans and machines
This addresses the issue of online misinformation by enhancing deepfake detection for human users and models, representing an incremental improvement through a novel human-centered approach.
The paper tackles the problem of deepfake detection by introducing a framework that amplifies artifacts in videos to make them more detectable by humans, resulting in greatly increased human detection across various conditions and also improving model accuracy and robustness.
Deepfakes can fuel online misinformation. As deepfakes get harder to recognize with the naked eye, human users become more reliant on deepfake detection models to help them decide whether a video is real or fake. Currently, models yield a prediction for a video's authenticity, but do not integrate a method for alerting a human user. We introduce a framework for amplifying artifacts in deepfake videos to make them more detectable by people. We propose a novel, semi-supervised Artifact Attention module, which is trained on human responses to create attention maps that highlight video artifacts, and magnify them to create a novel visual indicator we call "Deepfake Caricatures". In a user study, we demonstrate that Caricatures greatly increase human detection, across video presentation times and user engagement levels. We also introduce a deepfake detection model that incorporates the Artifact Attention module to increase its accuracy and robustness. Overall, we demonstrate the success of a human-centered approach to designing deepfake mitigation methods.