What Are LLMs Doing to Scientific Communication? Measuring Changes in Writing Practices and Reading Experience
For the NLP community, this work quantifies the impact of LLMs on scientific communication, revealing both objective changes in writing and subjective effects on reading experience.
The study measures changes in scientific writing practices due to LLMs by analyzing a corpus of 37,000+ NLP papers (2020-2024) and a synthetic dataset of 3,000 passages. Results show significant lexical and stylistic shifts, with LLM-modified texts rated as more understandable and exciting by experts, but also eliciting negative attitudes towards AI-assisted writing.
Has the style of scientific communication changed due to the growing use of large language models in the writing process? We address this question in the domain of Natural Language Processing by leveraging two data resources we create: a naturalistic corpus of over 37,000 papers from the ACL Anthology (2020-2024); and a synthetic dataset of 3,000 human-written passages and their LLM-generated improvements. We first implement a series of diachronic lexical analyses, showing that both word frequency and usage contexts have changed significantly over time, indicating semantic specialization in some cases and generalization in others. Broadening our perspective, we then model a range of more complex stylistic features and find that LLM-modified texts more frequently contain certain syntactic constructions, more complex and longer words and a lower lexical diversity. Finally, we connect these changes in writing practices to subjective reading experience through a pilot annotation study with 20 domain experts. They overall rate LLM-improved texts as more understandable and exciting, but also express negative qualitative attitudes towards LLMs, highlighting the strongly subjective effect of AI-assisted writing on reading experience.