Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learning
This addresses dataset biases in deepfake detection, a safety-critical application, by shifting to unsupervised methods to avoid shortcuts.
The paper identified that leading silence in audio-video deepfake datasets acts as a spurious feature, allowing near-perfect separation of real and fake samples, and proposed an unsupervised learning approach using real data to improve robustness.
Good datasets are essential for developing and benchmarking any machine learning system. Their importance is even more extreme for safety critical applications such as deepfake detection - the focus of this paper. Here we reveal that two of the most widely used audio-video deepfake datasets suffer from a previously unidentified spurious feature: the leading silence. Fake videos start with a very brief moment of silence and based on this feature alone, we can separate the real and fake samples almost perfectly. As such, previous audio-only and audio-video models exploit the presence of silence in the fake videos and consequently perform worse when the leading silence is removed. To circumvent latching on such unwanted artifact and possibly other unrevealed ones we propose a shift from supervised to unsupervised learning by training models exclusively on real data. We show that by aligning self-supervised audio-video representations we remove the risk of relying on dataset-specific biases and improve robustness in deepfake detection.