Sure! Here's a short and concise title for your paper: "Contamination in Generated Text Detection Benchmarks"
This work highlights a critical quality issue in datasets for AI-generated text detection, which is important for researchers and practitioners developing reliable detectors, though it is incremental as it focuses on improving an existing benchmark.
The paper addresses contamination in AI-generated text detection benchmarks, specifically showing that the DetectRL dataset contains simple patterns in 98.5% of Claude-LLM data, which detectors exploit as shortcuts, making them vulnerable to spoofing attacks; reprocessing the dataset with cleansing operations mitigates these attacks.
Large language models are increasingly used for many applications. To prevent illicit use, it is desirable to be able to detect AI-generated text. Training and evaluation of such detectors critically depend on suitable benchmark datasets. Several groups took on the tedious work of collecting, curating, and publishing large and diverse datasets for this task. However, it remains an open challenge to ensure high quality in all relevant aspects of such a dataset. For example, the DetectRL benchmark exhibits relatively simple patterns of AI-generation in 98.5% of the Claude-LLM data. These patterns may include introductory words such as "Sure! Here is the academic article abstract:", or instances where the LLM rejects the prompted task. In this work, we demonstrate that detectors trained on such data use such patterns as shortcuts, which facilitates spoofing attacks on the trained detectors. We consequently reprocessed the DetectRL dataset with several cleansing operations. Experiments show that such data cleansing makes direct attacks more difficult. The reprocessed dataset is publicly available.