LLM-Detector: Improving AI-Generated Chinese Text Detection with Open-Source LLM Instruction Tuning
This addresses the misuse of AI-generated texts by improving detection accuracy for Chinese content, though it is incremental as it builds on existing LLM and instruction tuning techniques.
The paper tackled the problem of poor out-of-domain detection and sentence-level performance in AI-generated Chinese text detection by proposing LLM-Detector, which uses instruction tuning on open-source LLMs, resulting in significant outperformance over baseline methods in both sentence-level and document-level detection with strong generalization capabilities.
ChatGPT and other general large language models (LLMs) have achieved remarkable success, but they have also raised concerns about the misuse of AI-generated texts. Existing AI-generated text detection models, such as based on BERT and RoBERTa, are prone to in-domain over-fitting, leading to poor out-of-domain (OOD) detection performance. In this paper, we first collected Chinese text responses generated by human experts and 9 types of LLMs, for which to multiple domains questions, and further created a dataset that mixed human-written sentences and sentences polished by LLMs. We then proposed LLM-Detector, a novel method for both document-level and sentence-level text detection through Instruction Tuning of LLMs. Our method leverages the wealth of knowledge LLMs acquire during pre-training, enabling them to detect the text they generate. Instruction tuning aligns the model's responses with the user's expected text detection tasks. Experimental results show that previous methods struggle with sentence-level AI-generated text detection and OOD detection. In contrast, our proposed method not only significantly outperforms baseline methods in both sentence-level and document-level text detection but also demonstrates strong generalization capabilities. Furthermore, since LLM-Detector is trained based on open-source LLMs, it is easy to customize for deployment.