A large-scale, unsupervised pipeline for automatic corpus annotation using LLMs: variation and change in the English consider construction
This addresses the problem of scalable data preparation for corpus-based research, though it is incremental as it builds on existing LLM capabilities.
The authors tackled the bottleneck of manual annotation in corpus linguistics by developing an unsupervised pipeline using large language models (LLMs) to automate grammatical annotation at scale, achieving over 98% accuracy on 143,933 sentences from the Corpus of Historical American English (COHA) in under 60 hours.
As natural language corpora expand at an unprecedented rate, manual annotation remains a significant methodological bottleneck in corpus linguistic work. We address this challenge by presenting a scalable, unsupervised pipeline for automating grammatical annotation in voluminous corpora using large language models (LLMs). Unlike previous supervised and iterative approaches, our method employs a four-phase workflow: prompt engineering, pre-hoc evaluation, automated batch processing, and post-hoc validation. We demonstrate the pipeline's accessibility and effectiveness through a diachronic case study of variation in the English consider construction. Using GPT-5 through the OpenAI API, we annotate 143,933 sentences from the Corpus of Historical American English (COHA) in under 60 hours, achieving 98%+ accuracy on two sophisticated annotation procedures. Our results suggest that LLMs can perform a range of data preparation tasks at scale with minimal human intervention, opening new possibilities for corpus-based research, though implementation requires attention to costs, licensing, and other ethical considerations.