Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits
This work addresses the problem of improving LLM-generated text quality and human-AI alignment for users in writing-intensive domains like social media, journalism, and education, though it is incremental by building on existing editing methods.
The study identified undesirable idiosyncrasies in LLM-generated text through a seven-category taxonomy and curated the LAMP corpus of 1,057 edited paragraphs, finding that none of the tested LLMs outperformed each other in writing quality. It also showed that automatic editing methods can improve alignment between LLM-generated and human-written text, as confirmed by expert preferences.
LLM-based applications are helping people write, and LLM-generated text is making its way into social media, journalism, and our classrooms. However, the differences between LLM-generated and human written text remain unclear. To explore this, we hired professional writers to edit paragraphs in several creative domains. We first found these writers agree on undesirable idiosyncrasies in LLM generated text, formalizing it into a seven-category taxonomy (e.g. clichés, unnecessary exposition). Second, we curated the LAMP corpus: 1,057 LLM-generated paragraphs edited by professional writers according to our taxonomy. Analysis of LAMP reveals that none of the LLMs used in our study (GPT4o, Claude-3.5-Sonnet, Llama-3.1-70b) outperform each other in terms of writing quality, revealing common limitations across model families. Third, building on existing work in automatic editing we evaluated methods to improve LLM-generated text. A large-scale preference annotation confirms that although experts largely prefer text edited by other experts, automatic editing methods show promise in improving alignment between LLM-generated and human-written text.