M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection
This addresses the challenge of identifying forged images for security and media integrity, though it is incremental as it builds on existing transformer and multi-modal approaches.
The paper tackles the problem of detecting Deepfakes by proposing M2TR, a multi-modal multi-scale transformer that captures subtle manipulation artifacts across spatial and frequency domains, and it outperforms state-of-the-art methods with clear margins while introducing a new dataset, SR-DF, with 4,000 videos.
The widespread dissemination of Deepfakes demands effective approaches that can detect perceptually convincing forged images. In this paper, we aim to capture the subtle manipulation artifacts at different scales using transformer models. In particular, we introduce a Multi-modal Multi-scale TRansformer (M2TR), which operates on patches of different sizes to detect local inconsistencies in images at different spatial levels. M2TR further learns to detect forgery artifacts in the frequency domain to complement RGB information through a carefully designed cross modality fusion block. In addition, to stimulate Deepfake detection research, we introduce a high-quality Deepfake dataset, SR-DF, which consists of 4,000 DeepFake videos generated by state-of-the-art face swapping and facial reenactment methods. We conduct extensive experiments to verify the effectiveness of the proposed method, which outperforms state-of-the-art Deepfake detection methods by clear margins.