Comparing LLM-generated and human-authored news text using formal syntactic theory
This provides insights into syntactic distinctions between human and LLM-generated text for the NYT genre, but it is incremental as it applies an existing method to new data.
The study compared the grammatical structure of New York Times-style text generated by six large language models against human-authored writing using Head-driven Phrase Structure Grammar, revealing systematic differences in HPSG grammar type distributions.
This study provides the first comprehensive comparison of New York Times-style text generated by six large language models against real, human-authored NYT writing. The comparison is based on a formal syntactic theory. We use Head-driven Phrase Structure Grammar (HPSG) to analyze the grammatical structure of the texts. We then investigate and illustrate the differences in the distributions of HPSG grammar types, revealing systematic distinctions between human and LLM-generated writing. These findings contribute to a deeper understanding of the syntactic behavior of LLMs as well as humans, within the NYT genre.