KLASSify to Verify: Audio-Visual Deepfake Detection Using SSL-based Audio and Handcrafted Visual Features
This work addresses the need for robust and interpretable deepfake detection for security applications, though it appears incremental as it builds on existing multimodal and SSL methods.
The paper tackled the problem of detecting and localizing audio-visual deepfakes, particularly under novel attack scenarios, by proposing a multimodal approach using handcrafted visual features and SSL-based audio features with graph attention networks, achieving an AUC of 92.78% for classification and an IoU of 0.3536 for temporal localization on the AV-Deepfake1M++ dataset.
The rapid development of audio-driven talking head generators and advanced Text-To-Speech (TTS) models has led to more sophisticated temporal deepfakes. These advances highlight the need for robust methods capable of detecting and localizing deepfakes, even under novel, unseen attack scenarios. Current state-of-the-art deepfake detectors, while accurate, are often computationally expensive and struggle to generalize to novel manipulation techniques. To address these challenges, we propose multimodal approaches for the AV-Deepfake1M 2025 challenge. For the visual modality, we leverage handcrafted features to improve interpretability and adaptability. For the audio modality, we adapt a self-supervised learning (SSL) backbone coupled with graph attention networks to capture rich audio representations, improving detection robustness. Our approach strikes a balance between performance and real-world deployment, focusing on resilience and potential interpretability. On the AV-Deepfake1M++ dataset, our multimodal system achieves AUC of 92.78% for deepfake classification task and IoU of 0.3536 for temporal localization using only the audio modality.