MIS-AVoiDD: Modality Invariant and Specific Representation for Audio-Visual Deepfake Detection
This work addresses the societal threat of deepfakes by improving detection for manipulated audio-visual content, representing an incremental advance in multimodal fusion techniques.
The paper tackles the challenge of detecting audio-visual deepfakes by addressing distributional gaps between audio and visual modalities, proposing a method that uses modality invariant and specific representations to improve fusion. The result is enhanced accuracy over state-of-the-art unimodal and multimodal detectors by 17.8% and 18.4%, respectively, achieving state-of-the-art performance.
Deepfakes are synthetic media generated using deep generative algorithms and have posed a severe societal and political threat. Apart from facial manipulation and synthetic voice, recently, a novel kind of deepfakes has emerged with either audio or visual modalities manipulated. In this regard, a new generation of multimodal audio-visual deepfake detectors is being investigated to collectively focus on audio and visual data for multimodal manipulation detection. Existing multimodal (audio-visual) deepfake detectors are often based on the fusion of the audio and visual streams from the video. Existing studies suggest that these multimodal detectors often obtain equivalent performances with unimodal audio and visual deepfake detectors. We conjecture that the heterogeneous nature of the audio and visual signals creates distributional modality gaps and poses a significant challenge to effective fusion and efficient performance. In this paper, we tackle the problem at the representation level to aid the fusion of audio and visual streams for multimodal deepfake detection. Specifically, we propose the joint use of modality (audio and visual) invariant and specific representations. This ensures that the common patterns and patterns specific to each modality representing pristine or fake content are preserved and fused for multimodal deepfake manipulation detection. Our experimental results on FakeAVCeleb and KoDF audio-visual deepfake datasets suggest the enhanced accuracy of our proposed method over SOTA unimodal and multimodal audio-visual deepfake detectors by $17.8$% and $18.4$%, respectively. Thus, obtaining state-of-the-art performance.