Performance Decay in Deepfake Detection: The Limitations of Training on Outdated Data
This addresses the critical issue of disinformation and fraud for society by highlighting the limitations of existing detection methods as deepfake technology advances, though it is incremental in proposing dataset curation and frame-level features as solutions.
The paper tackles the problem of deepfake detection performance decaying over time due to evolving generation techniques, showing that models trained on current data suffer a recall drop of over 30% when tested on deepfakes from just six months later. It introduces a two-stage detection method achieving an AUROC of over 99.8% on contemporary deepfakes.
The continually advancing quality of deepfake technology exacerbates the threats of disinformation, fraud, and harassment by making maliciously-generated synthetic content increasingly difficult to distinguish from reality. We introduce a simple yet effective two-stage detection method that achieves an AUROC of over 99.8% on contemporary deepfakes. However, this high performance is short-lived. We show that models trained on this data suffer a recall drop of over 30% when evaluated on deepfakes created with generation techniques from just six months later, demonstrating significant decay as threats evolve. Our analysis reveals two key insights for robust detection. Firstly, continued performance requires the ongoing curation of large, diverse datasets. Second, predictive power comes primarily from static, frame-level artifacts, not temporal inconsistencies. The future of effective deepfake detection therefore depends on rapid data collection and the development of advanced frame-level feature detectors.