From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detection
For researchers in audio-visual deepfake detection, this work highlights and tackles a previously overlooked domain shift problem, though the approach is incremental.
The paper addresses the domain shift in audio-visual deepfake detection when moving from talking to singing, where rhythmic vocalization weakens cross-modal coupling. They introduce the SHDF dataset and propose T-AVFD framework, achieving consistent improvements over baselines on multiple datasets.
With rapid advances in audio-visual generative models, reliable forgery detection becomes increasingly critical. Existing methods for audio-visual deepfake detection typically rely on cross-modal inconsistencies. In singing, rhythmic vocalization weakens this coupling and introduces a nontrivial domain shift, substantially degrading detection performance. We construct the Singing Head DeepFake (SHDF) dataset using rhythm-aware generative models to fill the gap in singing benchmarks. To cope with cross-scenario domain shifts, we propose a Text-guided Audio-Visual Forgery Detection (T-AVFD) framework that generalizes across both talking and singing scenarios. T-AVFD comprises a facial authenticity pattern learner and a multi-modal differential weight learning module. The pattern learner aligns facial features with multi-granularity textual descriptions to learn generalizable authenticity patterns. The weight learning module preserves intrinsic audio-visual consistency and adaptively integrates it with authenticity patterns via differential weighting. Extensive experiments on multiple talking head deepfake datasets and SHDF show consistent improvements over existing baselines and strong robustness under diverse perturbations.