EditLens: Quantifying the Extent of AI Editing in Text
This addresses the need for detecting AI editing in text, which has implications for authorship attribution, education, and policy, though it is incremental as it builds on prior work focused on AI-generated text detection.
The paper tackles the problem of distinguishing AI-edited text from human-written and AI-generated text, proposing EditLens, a model that quantifies the extent of AI editing and achieves state-of-the-art performance with F1 scores of 94.7% on binary and 90.4% on ternary classification tasks.
A significant proportion of queries to large language models ask them to edit user-provided text, rather than generate new text from scratch. While previous work focuses on detecting fully AI-generated text, we demonstrate that AI-edited text is distinguishable from human-written and AI-generated text. First, we propose using lightweight similarity metrics to quantify the magnitude of AI editing present in a text given the original human-written text and validate these metrics with human annotators. Using these similarity metrics as intermediate supervision, we then train EditLens, a regression model that predicts the amount of AI editing present within a text. Our model achieves state-of-the-art performance on both binary (F1=94.7%) and ternary (F1=90.4%) classification tasks in distinguishing human, AI, and mixed writing. Not only do we show that AI-edited text can be detected, but also that the degree of change made by AI to human writing can be detected, which has implications for authorship attribution, education, and policy. Finally, as a case study, we use our model to analyze the effects of AI-edits applied by Grammarly, a popular writing assistance tool. To encourage further research, we commit to publicly releasing our models and dataset.