The power of text similarity in identifying AI-LLM paraphrased documents: The case of BBC news articles and ChatGPT
This addresses the threat of copyright infringement and revenue loss for content creators by providing a detection tool for AI-paraphrased content, though it is incremental as it builds on existing similarity detection techniques.
The paper tackles the problem of detecting AI-paraphrased news articles, specifically those generated by ChatGPT, using a pattern-based similarity detection method. The method achieved detection accuracies of 96.23% for accuracy, 96.25% for precision, 96.21% for sensitivity, 96.25% for specificity, and 96.23% for F1 score on a dataset of 2,224 BBC articles and their ChatGPT paraphrases.
Generative AI paraphrased text can be used for copyright infringement and the AI paraphrased content can deprive substantial revenue from original content creators. Despite this recent surge of malicious use of generative AI, there are few academic publications that research this threat. In this article, we demonstrate the ability of pattern-based similarity detection for AI paraphrased news recognition. We propose an algorithmic scheme, which is not limited to detect whether an article is an AI paraphrase, but, more importantly, to identify that the source of infringement is the ChatGPT. The proposed method is tested with a benchmark dataset specifically created for this task that incorporates real articles from BBC, incorporating a total of 2,224 articles across five different news categories, as well as 2,224 paraphrased articles created with ChatGPT. Results show that our pattern similarity-based method, that makes no use of deep learning, can detect ChatGPT assisted paraphrased articles at percentages 96.23% for accuracy, 96.25% for precision, 96.21% for sensitivity, 96.25% for specificity and 96.23% for F1 score.