Unraveling Hidden Representations: A Multi-Modal Layer Analysis for Better Synthetic Content Forensics
This addresses the pressing need for robust fake detectors to combat misinformation from malicious users exploiting generative models, though it is incremental as it builds on existing multi-modal models.
The paper tackled the problem of detecting synthetic content like deepfakes by proposing a universal classifier using large pre-trained multi-modal models, achieving state-of-the-art results across audio and image modalities with computational efficiency and effectiveness in few-shot settings.
Generative models achieve remarkable results in multiple data domains, including images and texts, among other examples. Unfortunately, malicious users exploit synthetic media for spreading misinformation and disseminating deepfakes. Consequently, the need for robust and stable fake detectors is pressing, especially when new generative models appear everyday. While the majority of existing work train classifiers that discriminate between real and fake information, such tools typically generalize only within the same family of generators and data modalities, yielding poor results on other generative classes and data domains. Towards a universal classifier, we propose the use of large pre-trained multi-modal models for the detection of generative content. Effectively, we show that the latent code of these models naturally captures information discriminating real from fake. Building on this observation, we demonstrate that linear classifiers trained on these features can achieve state-of-the-art results across various modalities, while remaining computationally efficient, fast to train, and effective even in few-shot settings. Our work primarily focuses on fake detection in audio and images, achieving performance that surpasses or matches that of strong baseline methods.