TalkingHeadBench: A Multi-Modal Benchmark & Analysis of Talking-Head DeepFake Detection
This work addresses the need for up-to-date evaluation tools in deepfake detection to mitigate risks in media, politics, and finance, though it is incremental as it builds on existing detection methods.
The authors tackled the problem of outdated benchmarks for talking-head deepfake detection by introducing TalkingHeadBench, a comprehensive multi-modal benchmark and dataset that evaluates state-of-the-art detectors on advanced generators, revealing insights into model robustness and generalization.
The rapid advancement of talking-head deepfake generation fueled by advanced generative models has elevated the realism of synthetic videos to a level that poses substantial risks in domains such as media, politics, and finance. However, current benchmarks for deepfake talking-head detection fail to reflect this progress, relying on outdated generators and offering limited insight into model robustness and generalization. We introduce TalkingHeadBench, a comprehensive multi-model multi-generator benchmark and curated dataset designed to evaluate the performance of state-of-the-art detectors on the most advanced generators. Our dataset includes deepfakes synthesized by leading academic and commercial models and features carefully constructed protocols to assess generalization under distribution shifts in identity and generator characteristics. We benchmark a diverse set of existing detection methods, including CNNs, vision transformers, and temporal models, and analyze their robustness and generalization capabilities. In addition, we provide error analysis using Grad-CAM visualizations to expose common failure modes and detector biases. TalkingHeadBench is hosted on https://huggingface.co/datasets/luchaoqi/TalkingHeadBench with open access to all data splits and protocols. Our benchmark aims to accelerate research towards more robust and generalizable detection models in the face of rapidly evolving generative techniques.