'Your AI Text is not Mine': Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions
This work provides a foundational framework and benchmark for evaluating AI-generated text detection under realistic assumptions, addressing a critical need for researchers and practitioners in AI safety and content moderation.
The paper identifies a gap in AI-generated text detection research, where existing work often defines detection criteria loosely related to real-world needs. The authors propose systematic definitions of AI-generated text, introduce the AITDNA benchmark with detailed genesis annotations, and show that current detectors perform well only on specific notions but not broadly.
Although it is generally agreed that AI-generated text poses a broad societal risk, there is no common understanding in the AI-generated text detection literature on what constitutes harmful use. Rather, existing datasets and approaches often define their own criteria and make their own assumptions, sometimes implicitly, and often only loosely related to real-world needs and applications. To address this gap, we here systematically define various notions of AI-generated text and their characteristics. To study these, we collect AITDNA - a new benchmark of human-machine co-constructed texts that is annotated with detailed genesis information, such as the entire edit and AI-interaction history. We benchmark various machine-generated text detectors and find that they often only perform well for specific notions but not as broad detectors. We release code and data publicly.