Is ChatGPT Transforming Academics' Writing Style?
This research addresses the growing influence of large language models on academic writing, particularly in computer science, highlighting both positive and negative implications for scholars.
The study analyzed one million arXiv papers from 2018 to 2024 to assess the impact of ChatGPT on academic writing styles, finding that approximately 35% of computer science abstracts show signs of LLM-style text based on statistical word frequency changes.
Based on one million arXiv papers submitted from May 2018 to January 2024, we assess the textual density of ChatGPT's writing style in their abstracts through a statistical analysis of word frequency changes. Our model is calibrated and validated on a mixture of real abstracts and ChatGPT-modified abstracts (simulated data) after a careful noise analysis. The words used for estimation are not fixed but adaptive, including those with decreasing frequency. We find that large language models (LLMs), represented by ChatGPT, are having an increasing impact on arXiv abstracts, especially in the field of computer science, where the fraction of LLM-style abstracts is estimated to be approximately 35%, if we take the responses of GPT-3.5 to one simple prompt, "revise the following sentences", as a baseline. We conclude with an analysis of both positive and negative aspects of the penetration of LLMs into academics' writing style.