Human Bias in the Face of AI: Examining Human Judgment Against Text Labeled as AI Generated
This reveals a significant bias against AI in human judgment, with implications for undervaluing AI performance and improving human-AI collaboration, particularly in creative domains.
The study investigated how human bias affects perception of AI-generated versus human-generated text, finding that while raters could not distinguish them blindly, they strongly preferred content labeled as human by over 30%, even when labels were swapped.
As AI advances in text generation, human trust in AI generated content remains constrained by biases that go beyond concerns of accuracy. This study explores how bias shapes the perception of AI versus human generated content. Through three experiments involving text rephrasing, news article summarization, and persuasive writing, we investigated how human raters respond to labeled and unlabeled content. While the raters could not differentiate the two types of texts in the blind test, they overwhelmingly favored content labeled as "Human Generated," over those labeled "AI Generated," by a preference score of over 30%. We observed the same pattern even when the labels were deliberately swapped. This human bias against AI has broader societal and cognitive implications, as it undervalues AI performance. This study highlights the limitations of human judgment in interacting with AI and offers a foundation for improving human-AI collaboration, especially in creative fields.