Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement
This work critically examines the evaluation of syntactic abilities in neural language models, highlighting potential flaws in common benchmarks for researchers in NLP and linguistics.
The paper challenges the assumption that high accuracy on subject-verb agreement tasks indicates deep syntactic understanding in neural networks, showing that simple heuristics can achieve similar results, but finds that Transformers capture more grammatical structure than LSTMs in French object-verb agreement.
Many recent works have demonstrated that unsupervised sentence representations of neural networks encode syntactic information by observing that neural language models are able to predict the agreement between a verb and its subject. We take a critical look at this line of research by showing that it is possible to achieve high accuracy on this agreement task with simple surface heuristics, indicating a possible flaw in our assessment of neural networks' syntactic ability. Our fine-grained analyses of results on the long-range French object-verb agreement show that contrary to LSTMs, Transformers are able to capture a non-trivial amount of grammatical structure.