Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models
This provides insights into the internal workings of language models for researchers in NLP and linguistics, but it is incremental as it builds on existing syntactic evaluations.
The study applied causal mediation analysis to pre-trained neural language models to understand how they handle subject-verb agreement, finding that larger models do not necessarily learn stronger preferences and that different syntactic structures trigger distinct mechanisms.
Targeted syntactic evaluations have demonstrated the ability of language models to perform subject-verb agreement given difficult contexts. To elucidate the mechanisms by which the models accomplish this behavior, this study applies causal mediation analysis to pre-trained neural language models. We investigate the magnitude of models' preferences for grammatical inflections, as well as whether neurons process subject-verb agreement similarly across sentences with different syntactic structures. We uncover similarities and differences across architectures and model sizes -- notably, that larger models do not necessarily learn stronger preferences. We also observe two distinct mechanisms for producing subject-verb agreement depending on the syntactic structure of the input sentence. Finally, we find that language models rely on similar sets of neurons when given sentences with similar syntactic structure.