Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Model
This work addresses the problem of detecting deepfakes in videos for security and verification applications, presenting an incremental advancement by adapting foundation models for better generalization.
The paper tackles the challenge of generalizing deepfake detection to unseen forgery samples by proposing a side-network decoder that uses CLIP image encoder for spatial-temporal cues and introduces Facial Component Guidance to focus on key facial regions, achieving promising generalizability on challenging datasets with improvements in training data efficiency, parameter efficiency, and robustness.
Generative models have enabled the creation of highly realistic facial-synthetic images, raising significant concerns due to their potential for misuse. Despite rapid advancements in the field of deepfake detection, developing efficient approaches to leverage foundation models for improved generalizability to unseen forgery samples remains challenging. To address this challenge, we propose a novel side-network-based decoder that extracts spatial and temporal cues using the CLIP image encoder for generalized video-based Deepfake detection. Additionally, we introduce Facial Component Guidance (FCG) to enhance spatial learning generalizability by encouraging the model to focus on key facial regions. By leveraging the generic features of a vision-language foundation model, our approach demonstrates promising generalizability on challenging Deepfake datasets while also exhibiting superiority in training data efficiency, parameter efficiency, and model robustness.