Beyond Seeing Is Believing: On Crowdsourced Detection of Audiovisual Deepfakes
For researchers and platforms relying on human judgment to detect deepfakes, this work provides empirical evidence of the limitations and potential of crowdsourcing for audiovisual authenticity screening.
This paper evaluates crowd workers' ability to detect audiovisual deepfakes, finding that while they rarely misclassify authentic videos, they miss many manipulations and show limited agreement; aggregating judgments stabilizes authenticity but cannot recover consistently missed manipulations, and identifying manipulation types is substantially noisier.
Deepfakes are increasingly realistic and easy to produce, raising concerns about the reliability of human judgments in misinformation settings. We study audiovisual deepfake detection by measuring how consistently crowd workers distinguish authentic from manipulated videos and, when they flag a video as manipulated, how accurately they identify the manipulation type (audio-only, video-only, or audio-video) and how consistently they report manipulation timestamps. We run two matched crowdsourcing studies on Prolific using AV-Deepfake1M and the Trusted Media Challenge (TMC) dataset. We sample 48 videos per dataset (96 total) and collect 960 judgments (10 per video). Results show that crowd workers rarely misclassify authentic videos as manipulated, but they miss many manipulations, and agreement remains limited across videos. Aggregating multiple judgments per video stabilizes the authenticity signal, but it cannot recover manipulations that most workers consistently miss. Manipulation type identification is substantially noisier than authenticity detection even when workers detect a manipulation, with joint audio-video cases being particularly hard to recognize. Overall, these findings suggest that crowdsourcing can provide a scalable screening signal for audiovisual authenticity, while reliable modality attribution remains an open challenge.