Human-LLM Coevolution: Evidence from Academic Writing
This study provides evidence of human-LLM coevolution in academic writing, posing new challenges for detecting machine-generated text.
This paper analyzes arXiv abstracts and finds a significant drop in the frequency of words previously identified as overused by ChatGPT, such as "delve", after early 2024. Conversely, other ChatGPT-favored words like "significant" continued to increase, suggesting human adaptation to LLM use in academic writing.
With a statistical analysis of arXiv paper abstracts, we report a marked drop in the frequency of several words previously identified as overused by ChatGPT, such as "delve", starting soon after they were pointed out in early 2024. The frequency of certain other words favored by ChatGPT, such as "significant", has instead kept increasing. These phenomena suggest that some authors of academic papers have adapted their use of large language models (LLMs), for example, by selecting outputs or applying modifications to the LLM-generated content. Such coevolution and cooperation of humans and LLMs thus introduce additional challenges to the detection of machine-generated text in real-world scenarios. Estimating the impact of LLMs on academic writing by examining word frequency remains feasible, and more attention should be paid to words that were already frequently employed, including those that have decreased in frequency due to LLMs' disfavor.