Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?
This addresses the problem of detecting AI-generated text for users needing scalable detection methods, but it is incremental as it applies an existing model to a new task.
The paper investigates ChatGPT's zero-shot performance in detecting AI-generated versus human-written text, finding it can be leveraged in automated detection pipelines by focusing on specific aspects of the problem.
Large language models (LLMs) such as ChatGPT are increasingly being used for various use cases, including text content generation at scale. Although detection methods for such AI-generated text exist already, we investigate ChatGPT's performance as a detector on such AI-generated text, inspired by works that use ChatGPT as a data labeler or annotator. We evaluate the zero-shot performance of ChatGPT in the task of human-written vs. AI-generated text detection, and perform experiments on publicly available datasets. We empirically investigate if ChatGPT is symmetrically effective in detecting AI-generated or human-written text. Our findings provide insight on how ChatGPT and similar LLMs may be leveraged in automated detection pipelines by simply focusing on solving a specific aspect of the problem and deriving the rest from that solution. All code and data is available at https://github.com/AmritaBh/ChatGPT-as-Detector.