Do LLMs write like humans? Variation in grammatical and rhetorical styles
This research addresses the problem of distinguishing LLM output from human text for applications in detection and model evaluation, though it is incremental as it builds on prior work on surface features.
The study investigated whether large language models (LLMs) produce text with human-like grammatical and rhetorical styles, finding systematic differences in lexical, grammatical, and rhetorical features between LLM- and human-written texts, with larger and instruction-tuned models showing more pronounced variations.
Large language models (LLMs) are capable of writing grammatical text that follows instructions, answers questions, and solves problems. As they have advanced, it has become difficult to distinguish their output from human-written text. While past research has found some differences in surface features such as word choice and punctuation, and developed classifiers to detect LLM output, none has studied the rhetorical styles of LLMs. Using several variants of Llama 3 and GPT-4o, we construct two parallel corpora of human- and LLM-written texts from common prompts. Using Douglas Biber's set of lexical, grammatical, and rhetorical features, we identify systematic differences between LLMs and humans and between different LLMs. These differences persist when moving from smaller models to larger ones, and are larger for instruction-tuned models than base models. This observation of differences demonstrates that despite their advanced abilities, LLMs struggle to match human stylistic variation. Attention to more advanced linguistic features can hence detect patterns in their behavior not previously recognized.