Domain Regeneration: How well do LLMs match syntactic properties of text domains?
This work addresses the problem of evaluating LLM fidelity to domain-specific syntactic features for researchers and practitioners, but it is incremental as it applies existing observational methods to new analyses.
The study investigated how well large language models (LLMs) approximate syntactic properties of text domains like Wikipedia and news, finding that regenerated text often shows shifted means, lower standard deviation, and reduced long tails compared to human originals.
Recent improvement in large language model performance have, in all likelihood, been accompanied by improvement in how well they can approximate the distribution of their training data. In this work, we explore the following question: which properties of text domains do LLMs faithfully approximate, and how well do they do so? Applying observational approaches familiar from corpus linguistics, we prompt a commonly used, opensource LLM to regenerate text from two domains of permissively licensed English text which are often contained in LLM training data -- Wikipedia and news text. This regeneration paradigm allows us to investigate whether LLMs can faithfully match the original human text domains in a fairly semantically-controlled setting. We investigate varying levels of syntactic abstraction, from more simple properties like sentence length, and article readability, to more complex and higher order properties such as dependency tag distribution, parse depth, and parse complexity. We find that the majority of the regenerated distributions show a shifted mean, a lower standard deviation, and a reduction of the long tail, as compared to the human originals.