Real-Aware Residual Model Merging for Deepfake Detection
This addresses the problem of rapidly evolving deepfake generators for detection systems, offering a composable solution to avoid costly retraining, though it is incremental as it builds on model merging techniques.
The paper tackles the challenge of deepfake detection by proposing Real-aware Residual Model Merging (R^2M), a training-free framework that merges specialist models to improve performance without exhaustive retraining, achieving better results than joint training and baselines across in-distribution, cross-dataset, and unseen-dataset evaluations.
Deepfake generators evolve quickly, making exhaustive data collection and repeated retraining impractical. We argue that model merging is a natural fit for deepfake detection: unlike generic multi-task settings with disjoint labels, deepfake specialists share the same binary decision and differ in generator-specific artifacts. Empirically, we show that simple weight averaging preserves Real representations while attenuating Fake-specific cues. Building upon these findings, we propose Real-aware Residual Model Merging (R$^2$M), a training-free parameter-space merging framework. R$^2$M estimates a shared Real component via a low-rank factorization of task vectors, decomposes each specialist into a Real-aligned part and a Fake residual, denoises residuals with layerwise rank truncation, and aggregates them with per-task norm matching to prevent any single generator from dominating. A concise rationale explains why a simple head suffices: the Real component induces a common separation direction in feature space, while truncated residuals contribute only minor off-axis variations. Across in-distribution, cross-dataset, and unseen-dataset, R$^2$M outperforms joint training and other merging baselines. Importantly, R$^2$M is also composable: when a new forgery family appears, we fine-tune one specialist and re-merge, eliminating the need for retraining.