Teaching Syntax by Adversarial Distraction
This addresses the issue of syntax awareness in NLP models for researchers and practitioners, but it is incremental as it builds on existing datasets and methods.
The authors tackled the problem that existing entailment datasets allow models to ignore syntax, by introducing synthetic datasets to teach grammar and word order. They found that without retraining, popular models fail to recognize syntactic differences, and with retraining, only some models learn to compare syntax properly.
Existing entailment datasets mainly pose problems which can be answered without attention to grammar or word order. Learning syntax requires comparing examples where different grammar and word order change the desired classification. We introduce several datasets based on synthetic transformations of natural entailment examples in SNLI or FEVER, to teach aspects of grammar and word order. We show that without retraining, popular entailment models are unaware that these syntactic differences change meaning. With retraining, some but not all popular entailment models can learn to compare the syntax properly.