Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore
This addresses the need for reliable, data-free detection of AI-generated content, which is crucial for applications like content moderation and academic integrity, though it is an incremental improvement over existing zero-shot approaches.
The paper tackles the problem of detecting LLM-generated text without training data by proposing a black-box zero-shot method based on grammatical error scores, achieving an average AUROC of 98.62% and outperforming existing state-of-the-art methods.
The efficacy of detectors for texts generated by large language models (LLMs) substantially depends on the availability of large-scale training data. However, white-box zero-shot detectors, which require no such data, are limited by the accessibility of the source model of the LLM-generated text. In this paper, we propose a simple yet effective black-box zero-shot detection approach based on the observation that, from the perspective of LLMs, human-written texts typically contain more grammatical errors than LLM-generated texts. This approach involves calculating the Grammar Error Correction Score (GECScore) for the given text to differentiate between human-written and LLM-generated text. Experimental results show that our method outperforms current state-of-the-art (SOTA) zero-shot and supervised methods, achieving an average AUROC of 98.62% across XSum and Writing Prompts dataset. Additionally, our approach demonstrates strong reliability in the wild, exhibiting robust generalization and resistance to paraphrasing attacks. Data and code are available at: https://github.com/NLP2CT/GECScore.