Assessing BERT's Syntactic Abilities
This work addresses the problem of understanding syntactic capabilities in large language models for NLP researchers, but it is incremental as it evaluates an existing model without introducing new methods.
The study assessed BERT's ability to capture English syntactic phenomena, such as subject-verb agreement and reflexive anaphora, using various stimuli, and found it performed remarkably well across all cases.
I assess the extent to which the recently introduced BERT model captures English syntactic phenomena, using (1) naturally-occurring subject-verb agreement stimuli; (2) "coloreless green ideas" subject-verb agreement stimuli, in which content words in natural sentences are randomly replaced with words sharing the same part-of-speech and inflection; and (3) manually crafted stimuli for subject-verb agreement and reflexive anaphora phenomena. The BERT model performs remarkably well on all cases.