Are Large Language Models the future crowd workers of Linguistics?
This research tackles the problem of inefficient and potentially unreliable data collection methods in empirical linguistic research, offering a potentially more efficient and reliable alternative for linguists and researchers in the humanities.
This study investigates the potential of Large Language Models (LLMs) to replace human participants in empirical linguistic research, with results showing that LLMs can outperform human informants in certain tasks. The study found that LLMs, specifically OpenAI's GPT-4o-mini model, can achieve high performance with zero-shot prompting and even higher alignment with human performance using Chain-of-Thought (CoT) prompting.
Data elicitation from human participants is one of the core data collection strategies used in empirical linguistic research. The amount of participants in such studies may vary considerably, ranging from a handful to crowdsourcing dimensions. Even if they provide resourceful extensive data, both of these settings come alongside many disadvantages, such as low control of participants' attention during task completion, precarious working conditions in crowdsourcing environments, and time-consuming experimental designs. For these reasons, this research aims to answer the question of whether Large Language Models (LLMs) may overcome those obstacles if included in empirical linguistic pipelines. Two reproduction case studies are conducted to gain clarity into this matter: Cruz (2023) and Lombard et al. (2021). The two forced elicitation tasks, originally designed for human participants, are reproduced in the proposed framework with the help of OpenAI's GPT-4o-mini model. Its performance with our zero-shot prompting baseline shows the effectiveness and high versatility of LLMs, that tend to outperform human informants in linguistic tasks. The findings of the second replication further highlight the need to explore additional prompting techniques, such as Chain-of-Thought (CoT) prompting, which, in a second follow-up experiment, demonstrates higher alignment to human performance on both critical and filler items. Given the limited scale of this study, it is worthwhile to further explore the performance of LLMs in empirical Linguistics and in other future applications in the humanities.