Environmental Sound Deepfake Detection Using Deep-Learning Framework
This work addresses the emerging problem of environmental sound deepfake detection for audio forensics, but the approach is incremental as it applies existing fine-tuning techniques to a new domain.
The paper proposes a deep-learning framework for detecting deepfake environmental sounds, achieving high performance (Accuracy 0.98, F1 0.95, AuC 0.99 on EnvSDD; Accuracy 0.88, F1 0.77, AuC 0.92 on ESDD-Challenge-TestSet) by fine-tuning a pre-trained WavLM model with a three-stage training strategy.
In this paper, we propose a deep-learning framework for environmental sound deepfake detection (ESDD) -- the task of identifying whether the sound scene and sound event in an input audio recording is fake or not. To this end, we conducted extensive experiments to explore how individual spectrograms, a wide range of network architectures and pre-trained models, ensemble of spectrograms or network architectures affect the ESDD task performance. The experimental results on the benchmark datasets of EnvSDD and ESDD-Challenge-TestSet indicate that detecting deepfake audio of sound scene and detecting deepfake audio of sound event should be considered as individual tasks. We also indicate that the approach of finetuning a pre-trained model is more effective compared with training a model from scratch for the ESDD task. Eventually, our best model, which was finetuned from the pre-trained WavLM model with the proposed three-stage training strategy, achieve the Accuracy of 0.98, F1 Score of 0.95, AuC of 0.99 on EnvSDD Test subset and the Accuracy of 0.88, F1 Score of 0.77, and AuC of 0.92 on ESDD-Challenge-TestSet dataset.