Using Machine Learning to Distinguish Human-written from Machine-generated Creative Fiction
This addresses the threat of generative AI to creative writers and literary culture by providing a tool for editors and publishers to detect deceptive text, though it is incremental as it builds on existing detection methods applied to a new domain.
The study tackled the problem of distinguishing human-written from machine-generated creative fiction, specifically in classic detective novels, by training machine learning classifiers that achieved over 95% accuracy, significantly outperforming human judges who scored below 55%.
Following the universal availability of generative AI systems with the release of ChatGPT, automatic detection of deceptive text created by Large Language Models has focused on domains such as academic plagiarism and "fake news". However, generative AI also poses a threat to the livelihood of creative writers, and perhaps to literary culture in general, through reduction in quality of published material. Training a Large Language Model on writers' output to generate "sham books" in a particular style seems to constitute a new form of plagiarism. This problem has been little researched. In this study, we trained Machine Learning classifier models to distinguish short samples of human-written from machine-generated creative fiction, focusing on classic detective novels. Our results show that a Naive Bayes and a Multi-Layer Perceptron classifier achieved a high degree of success (accuracy > 95%), significantly outperforming human judges (accuracy < 55%). This approach worked well with short text samples (around 100 words), which previous research has shown to be difficult to classify. We have deployed an online proof-of-concept classifier tool, AI Detective, as a first step towards developing lightweight and reliable applications for use by editors and publishers, with the aim of protecting the economic and cultural contribution of human authors.