LLM or Human? Perceptions of Trust and Information Quality in Research Summaries
This addresses the problem of trust and quality perceptions in AI-assisted scientific writing for researchers and policymakers, providing insights into evolving norms, though it is incremental as it builds on existing concerns about AI in academia.
This paper investigated whether readers with ML expertise can distinguish between human- and LLM-generated research abstracts and how perceptions of LLM involvement affect evaluations of quality and trustworthiness, finding that participants struggle to reliably identify LLM-generated content but their beliefs significantly shape ratings, with LLM-edited abstracts rated more favorably than human-only or LLM-only ones.
Large Language Models (LLMs) are increasingly used to generate and edit scientific abstracts, yet their integration into academic writing raises questions about trust, quality, and disclosure. Despite growing adoption, little is known about how readers perceive LLM-generated summaries and how these perceptions influence evaluations of scientific work. This paper presents a mixed-methods survey experiment investigating whether readers with ML expertise can distinguish between human- and LLM-generated abstracts, how actual and perceived LLM involvement affects judgments of quality and trustworthiness, and what orientations readers adopt toward AI-assisted writing. Our findings show that participants struggle to reliably identify LLM-generated content, yet their beliefs about LLM involvement significantly shape their evaluations. Notably, abstracts edited by LLMs are rated more favorably than those written solely by humans or LLMs. We also identify three distinct reader orientations toward LLM-assisted writing, offering insights into evolving norms and informing policy around disclosure and acceptable use in scientific communication.