Users Favor LLM-Generated Content -- Until They Know It's AI
This addresses the societal challenge of AI perception in quality assessments, but it is incremental as it builds on existing research about human-AI interaction biases.
The study investigated how people evaluate responses to questions when the source (human or AI) is hidden or revealed, finding that participants generally prefer AI-generated responses, but this preference drops significantly when the AI origin is disclosed, indicating bias against AI content.
In this paper, we investigate how individuals evaluate human and large langue models generated responses to popular questions when the source of the content is either concealed or disclosed. Through a controlled field experiment, participants were presented with a set of questions, each accompanied by a response generated by either a human or an AI. In a randomized design, half of the participants were informed of the response's origin while the other half remained unaware. Our findings indicate that, overall, participants tend to prefer AI-generated responses. However, when the AI origin is revealed, this preference diminishes significantly, suggesting that evaluative judgments are influenced by the disclosure of the response's provenance rather than solely by its quality. These results underscore a bias against AI-generated content, highlighting the societal challenge of improving the perception of AI work in contexts where quality assessments should be paramount.