Evade ChatGPT Detectors via A Single Space
This work exposes vulnerabilities in ChatGPT detectors, posing a problem for misuse prevention, and is incremental in revealing specific evasion tactics.
The paper challenges the assumption that distributional gaps exist between human and AI-generated text, finding that detectors rely on subtle differences like extra spaces, and proposes the SpaceInfi strategy to evade detection, which is effective across multiple benchmarks and detectors.
ChatGPT brings revolutionary social value but also raises concerns about the misuse of AI-generated text. Consequently, an important question is how to detect whether texts are generated by ChatGPT or by human. Existing detectors are built upon the assumption that there are distributional gaps between human-generated and AI-generated text. These gaps are typically identified using statistical information or classifiers. Our research challenges the distributional gap assumption in detectors. We find that detectors do not effectively discriminate the semantic and stylistic gaps between human-generated and AI-generated text. Instead, the "subtle differences", such as an extra space, become crucial for detection. Based on this discovery, we propose the SpaceInfi strategy to evade detection. Experiments demonstrate the effectiveness of this strategy across multiple benchmarks and detectors. We also provide a theoretical explanation for why SpaceInfi is successful in evading perplexity-based detection. And we empirically show that a phenomenon called token mutation causes the evasion for language model-based detectors. Our findings offer new insights and challenges for understanding and constructing more applicable ChatGPT detectors.