PAWS: Paraphrase Adversaries from Word Scrambling
This addresses a critical bottleneck for NLP researchers and practitioners by providing a dataset that exposes weaknesses in existing models and drives progress in understanding context and structure.
The paper tackled the problem of paraphrase identification models failing on sentence pairs with high lexical overlap but different meanings by introducing the PAWS dataset, which improved model accuracy from below 40% to 85% on this challenge while maintaining performance on other tasks.
Existing paraphrase identification datasets lack sentence pairs that have high lexical overlap without being paraphrases. Models trained on such data fail to distinguish pairs like flights from New York to Florida and flights from Florida to New York. This paper introduces PAWS (Paraphrase Adversaries from Word Scrambling), a new dataset with 108,463 well-formed paraphrase and non-paraphrase pairs with high lexical overlap. Challenging pairs are generated by controlled word swapping and back translation, followed by fluency and paraphrase judgments by human raters. State-of-the-art models trained on existing datasets have dismal performance on PAWS (<40% accuracy); however, including PAWS training data for these models improves their accuracy to 85% while maintaining performance on existing tasks. In contrast, models that do not capture non-local contextual information fail even with PAWS training examples. As such, PAWS provides an effective instrument for driving further progress on models that better exploit structure, context, and pairwise comparisons.