Transforming Chatbot Text: A Sequence-to-Sequence Approach
This work addresses the challenge of distinguishing AI-generated from human text for security and detection applications, but it is incremental as it builds on existing detection and transformation methods.
The paper tackles the problem of detecting AI-generated text by adversarially transforming GPT-generated text using sequence-to-sequence models to make it more human-like, resulting in significantly reduced accuracy for classification models on the modified text, though retraining these models restores high accuracy.
Due to advances in Large Language Models (LLMs) such as ChatGPT, the boundary between human-written text and AI-generated text has become blurred. Nevertheless, recent work has demonstrated that it is possible to reliably detect GPT-generated text. In this paper, we adopt a novel strategy to adversarially transform GPT-generated text using sequence-to-sequence (Seq2Seq) models, with the goal of making the text more human-like. We experiment with the Seq2Seq models T5-small and BART which serve to modify GPT-generated sentences to include linguistic, structural, and semantic components that may be more typical of human-authored text. Experiments show that classification models trained to distinguish GPT-generated text are significantly less accurate when tested on text that has been modified by these Seq2Seq models. However, after retraining classification models on data generated by our Seq2Seq technique, the models are able to distinguish the transformed GPT-generated text from human-generated text with high accuracy. This work adds to the accumulating knowledge of text transformation as a tool for both attack -- in the sense of defeating classification models -- and defense -- in the sense of improved classifiers -- thereby advancing our understanding of AI-generated text.