Proteus: Automated Adversarial Robustness Testing for Audio Deepfake Detectors
For developers of audio deepfake detectors, this provides a systematic method to identify and fix vulnerabilities, though the approach is incremental and domain-specific.
Proteus automates robustness testing of audio deepfake detectors by searching for sequences of everyday audio transformations that fool the detector while preserving speech quality, and uses findings to harden the detector via targeted retraining.
We present Proteus, a framework developed at Resemble AI for automated robustness testing of our audio deepfake detection system. Given a detector, Proteus systematically searches over sequences of everyday audio transformations (codec transcoding, additive noise, reverberation, dynamic-range compression, and VoIP simulation) to find combinations that fool the detector while preserving speech quality. We propose two complementary search strategies: (1) a breadth-first search that exhaustively maps augmentation effectiveness across the parameter space, and (2) a Q-learning agent designed to efficiently discover deeper attack chains by exploiting structural patterns in the BFS data. We report findings from continuous deployment of Proteus against our production detector, showing that specific augmentation chains can reliably flip detection verdicts while preserving speech intelligibility and speaker identity. We discuss how these findings are used to harden the detector through targeted retraining.