Detecting LLM-assisted writing in scientific communication: Are we there yet?
This addresses the problem of insufficient transparency in LLM-assisted writing for scientific communities, but it is incremental as it builds on existing detection methods.
The paper evaluated four state-of-the-art LLM-generated text detectors and found they performed poorly compared to a simple ad-hoc detector that identifies abrupt writing style changes, highlighting the need for specialized detectors to improve acknowledgment of LLM use in scientific communication.
Large Language Models (LLMs), exemplified by ChatGPT, have significantly reshaped text generation, particularly in the realm of writing assistance. While ethical considerations underscore the importance of transparently acknowledging LLM use, especially in scientific communication, genuine acknowledgment remains infrequent. A potential avenue to encourage accurate acknowledging of LLM-assisted writing involves employing automated detectors. Our evaluation of four cutting-edge LLM-generated text detectors reveals their suboptimal performance compared to a simple ad-hoc detector designed to identify abrupt writing style changes around the time of LLM proliferation. We contend that the development of specialized detectors exclusively dedicated to LLM-assisted writing detection is necessary. Such detectors could play a crucial role in fostering more authentic recognition of LLM involvement in scientific communication, addressing the current challenges in acknowledgment practices.