What Counts as Real? Speech Restoration and Voice Quality Conversion Pose New Challenges to Deepfake Detection
This addresses a critical flaw in deepfake detection for audio security, highlighting that current binary classifiers fail under layered generative processing, which is an incremental but important refinement.
The paper tackles the problem that audio anti-spoofing systems misclassify benign speech transformations like voice conversion and restoration as spoofing, showing that these transformations cause distributional shifts in self-supervised learning embeddings and reduce classifier separability. It finds that reformulating anti-spoofing as a multi-class problem improves robustness to such shifts while maintaining spoof detection.
Audio anti-spoofing systems are typically formulated as binary classifiers distinguishing bona fide from spoofed speech. This assumption fails under layered generative processing, where benign transformations introduce distributional shifts that are misclassified as spoofing. We show that phonation-modifying voice conversion and speech restoration are treated as out-of-distribution despite preserving speaker authenticity. Using a multi-class setup separating bona fide, converted, spoofed, and converted-spoofed speech, we analyse model behaviour through self-supervised learning (SSL) embeddings and acoustic correlates. The benign transformations induce a drift in the SSL space, compressing bona fide and spoofed speech and reducing classifier separability. Reformulating anti-spoofing as a multi-class problem improves robustness to benign shifts while preserving spoof detection, suggesting binary systems model the distribution of raw speech rather than authenticity itself.