What Makes Language Models Good-enough?
This work addresses the problem of making language models more efficient and human-like in processing for NLP researchers, but it is incremental as it builds on existing psycholinguistic concepts and Transformer architectures.
The study investigated what architectural features in Transformers, such as the number of layers and self-attention heads, enable language models to learn human-like 'good-enough' language processing, finding that models with fewer layers and/or heads can achieve this performance on a new evaluation dataset of 7,680 examples.
Psycholinguistic research suggests that humans may build a representation of linguistic input that is 'good-enough' for the task at hand. This study examines what architectural features make language models learn human-like good-enough language processing. We focus on the number of layers and self-attention heads in Transformers. We create a good-enough language processing (GELP) evaluation dataset (7,680 examples), which is designed to test the effects of two plausibility types, eight construction types, and three degrees of memory cost on language processing. To annotate GELP, we first conduct a crowdsourcing experiment whose design follows prior psycholinguistic studies. Our model evaluation against the annotated GELP then reveals that the full model as well as models with fewer layers and/or self-attention heads exhibit a good-enough performance. This result suggests that models with shallower depth and fewer heads can learn good-enough language processing.