A Scalable Pipeline for Estimating Verb Frame Frequencies Using Large Language Models
This work addresses the need for scalable and accurate VFF estimation for researchers in linguistics and psycholinguistics, though it is incremental as it builds on existing LLM capabilities.
The researchers tackled the problem of estimating Verb Frame Frequencies (VFFs) by developing an automated pipeline using large language models, which outperformed existing syntactic parsers and enabled the creation of a new VFF database with broader coverage and finer-grained distinctions.
We present an automated pipeline for estimating Verb Frame Frequencies (VFFs), the frequency with which a verb appears in particular syntactic frames. VFFs provide a powerful window into syntax in both human and machine language systems, but existing tools for calculating them are limited in scale, accuracy, or accessibility. We use large language models (LLMs) to generate a corpus of sentences containing 476 English verbs. Next, by instructing an LLM to behave like an expert linguist, we had it analyze the syntactic structure of the sentences in this corpus. This pipeline outperforms two widely used syntactic parsers across multiple evaluation datasets. Furthermore, it requires far fewer resources than manual parsing (the gold-standard), thereby enabling rapid, scalable VFF estimation. Using the LLM parser, we produce a new VFF database with broader verb coverage, finer-grained syntactic distinctions, and explicit estimates of the relative frequencies of structural alternates commonly studied in psycholinguistics. The pipeline is easily customizable and extensible to new verbs, syntactic frames, and even other languages. We present this work as a proof of concept for automated frame frequency estimation, and release all code and data to support future research.