Humans Cannot Detect AI-Generated Media But Communities May -- For Now: Collaborative AI Detection in r/RealOrAI on Reddit
For researchers and platforms concerned with AI-generated media detection, this work provides a large-scale, naturalistic study of human detection behavior, revealing the limits of heuristic-based detection and the potential of community aggregation.
This paper analyzes a year of activity from the Reddit community r/RealOrAI, finding that community detection of AI-generated media reaches 72% accuracy but with a growing false-positive bias. Perceptual features dominate individual reasoning (70%), while provenance verification is rare (4%) but amplified 4.3x in community summaries, showing aggregation improves reliability.
We study human AI-detection behaviour at scale using a year of activity from r/RealOrAI, a Reddit community where users collaboratively assess whether visual media is real or AI-generated. The community is moderated by a bot that solicits verified labels from submitters of self-challenging "[GUESS]" posts and publishes an aggregate community prediction for each post, yielding naturalistic ground truth at scale. Community detection accuracy reaches 72% on [GUESS] posts with a systematic false-positive bias that intensifies over the year as the community's AI-suspicion grows. Using a six-LLM ensemble validated against human-annotated ground truth, we classify 10k reasoning-bearing comments along six cues covering perceptual features, context, consistency, AI knowledge, subject-matter expertise and provenance (tracing the media to its source). Perceptual features (scene, visual artifacts, anatomy physics, lighting, behavior, text, audio) dominate reasoning (70%) while provenance verification is rarest (4%) at the individual level but is amplified 4.3x in community summaries, revealing aggregation as a reliability filter that selectively surfaces diagnostic evidence. These findings reveal the limits of heuristic-based detection and show how online communities collectively navigate an increasingly contested information environment.