Alethia: A Foundational Encoder for Voice Deepfakes
For researchers and practitioners in audio forensics, this provides a new pretraining paradigm that overcomes diminishing returns of finetuning speech foundation models, achieving strong performance across diverse deepfake detection tasks.
The authors propose Alethia, the first foundational audio encoder for voice deepfake detection and localization, which combines bottleneck masked embedding prediction with flow-matching based spectrogram reconstruction. It significantly outperforms state-of-the-art speech foundation models across 56 benchmark datasets on 5 tasks, with superior robustness and zero-shot generalization to unseen domains like singing deepfakes.
Existing voice deepfake detection and localization models rely heavily on representations extracted from speech foundation models (SFMs). However, downstream finetuning has now reached a state of diminishing returns. In this paper, we shift the focus to pretraining and propose a novel recipe that combines bottleneck masked embedding prediction with flow-matching based spectrogram reconstruction. The outcome, Alethia, is the first foundational audio encoder for various voice deepfake detection and localization tasks. We evaluate on $5$ different tasks with $56$ benchmark datasets, and note Alethia significantly outperforms state-of-the-art SFMs with superior robustness to real-world perturbations and zero-shot generalization to unseen domains (e.g., singing deepfakes). We also demonstrate the limitation of discrete targets in masked token prediction, and show the importance of continuous embedding prediction and generative pretraining for capturing deepfake artifacts.