AuViRe: Audio-visual Speech Representation Reconstruction for Deepfake Temporal Localization
This addresses the integrity of digital media against subtle malicious manipulations, with incremental advancements in detection accuracy.
The paper tackles the problem of temporal localization of deepfakes in audio-visual content by reconstructing speech representations across modalities, achieving improvements such as +8.9 AP@0.95 on LAV-DF and +9.6 AP@0.5 on AV-Deepfake1M.
With the rapid advancement of sophisticated synthetic audio-visual content, e.g., for subtle malicious manipulations, ensuring the integrity of digital media has become paramount. This work presents a novel approach to temporal localization of deepfakes by leveraging Audio-Visual Speech Representation Reconstruction (AuViRe). Specifically, our approach reconstructs speech representations from one modality (e.g., lip movements) based on the other (e.g., audio waveform). Cross-modal reconstruction is significantly more challenging in manipulated video segments, leading to amplified discrepancies, thereby providing robust discriminative cues for precise temporal forgery localization. AuViRe outperforms the state of the art by +8.9 AP@0.95 on LAV-DF, +9.6 AP@0.5 on AV-Deepfake1M, and +5.1 AUC on an in-the-wild experiment. Code available at https://github.com/mever-team/auvire.