A logical-based corpus for cross-lingual evaluation
This addresses the need for better evaluation benchmarks in NLP to assess model understanding beyond simple patterns, though it is incremental as it builds on existing logical form tasks.
The authors tackled the problem of evaluating deep learning models' true capacity for textual inference by creating a new dataset focused on contradiction detection requiring logical forms, and found that while BERT generalizes well, it struggles with counting operators, with cross-lingual transfer demonstrated between English and Portuguese.
At present, different deep learning models are presenting high accuracy on popular inference datasets such as SNLI, MNLI, and SciTail. However, there are different indicators that those datasets can be exploited by using some simple linguistic patterns. This fact poses difficulties to our understanding of the actual capacity of machine learning models to solve the complex task of textual inference. We propose a new set of syntactic tasks focused on contradiction detection that require specific capacities over linguistic logical forms such as: Boolean coordination, quantifiers, definite description, and counting operators. We evaluate two kinds of deep learning models that implicitly exploit language structure: recurrent models and the Transformer network BERT. We show that although BERT is clearly more efficient to generalize over most logical forms, there is space for improvement when dealing with counting operators. Since the syntactic tasks can be implemented in different languages, we show a successful case of cross-lingual transfer learning between English and Portuguese.