Process-Oriented Evaluation of AI-Assisted Scientific Writing
For researchers and practitioners in scientific writing and AI, this work provides a process-oriented evaluation of editing behaviors and limitations of AI-generated text.
This study evaluates how humans revise AI-generated vs. human-authored scientific abstracts, analyzing 869 keystroke-level edit logs. It finds that AI abstracts have higher sentence-level agency but lower global coherence, and that experts switch from restructuring to substitution when AI source is disclosed, while language models still struggle with global coherence.
Bad writing hinders the publication of science. The role of artificial intelligence (AI) in generating and editing scientific texts remains unsettled. Abstracts serve as the critical gateway to scientific manuscripts, often shaping readers' interest. We inspect how individuals revise AI-generated abstracts compared to human-authored abstracts when incentivized to communicate scientific content. Using 869 keystroke-level edit logs with 240k total edits, we construct behavioral labels and measure linguistic properties of edit bursts to investigate the edit trajectories. AI abstracts exhibit higher sentence-level agency, whereas human-authored abstracts outperform in global coherence, even with edits. Experts engage in stigmatic behavior, switching their strategy from predominantly restructuring to substitution when AI source is disclosed. Language Models (LMs) improve edit outcomes through a mix of local and global features, but still actively struggle with global coherence. Both humans and LMs often target the weakest sections of abstracts, but fail to improve stronger areas. Our large-scale process-oriented evaluation highlights the perks and pitfalls of both human and LM editing processes as machine-generated texts emerge in scientific communication.