Exploring Self-Supervised Vision Transformers for Deepfake Detection: A Comparative Analysis
This addresses the deepfake detection community's need for efficient and generalizable methods, though it is incremental as it builds on existing SSL techniques.
This paper tackled the problem of detecting facial deepfake images and videos by comparing self-supervised pre-trained vision transformers (ViTs) to supervised ViTs and ConvNets, finding that SSL ViTs offer comparable adaptability and explainability with modest data and resource-efficient partial fine-tuning.
This paper investigates the effectiveness of self-supervised pre-trained vision transformers (ViTs) compared to supervised pre-trained ViTs and conventional neural networks (ConvNets) for detecting facial deepfake images and videos. It examines their potential for improved generalization and explainability, especially with limited training data. Despite the success of transformer architectures in various tasks, the deepfake detection community is hesitant to use large ViTs as feature extractors due to their perceived need for extensive data and suboptimal generalization with small datasets. This contrasts with ConvNets, which are already established as robust feature extractors. Additionally, training ViTs from scratch requires significant resources, limiting their use to large companies. Recent advancements in self-supervised learning (SSL) for ViTs, like masked autoencoders and DINOs, show adaptability across diverse tasks and semantic segmentation capabilities. By leveraging SSL ViTs for deepfake detection with modest data and partial fine-tuning, we find comparable adaptability to deepfake detection and explainability via the attention mechanism. Moreover, partial fine-tuning of ViTs is a resource-efficient option.