An Analysis of Negation in Natural Language Understanding Corpora
This addresses a data quality issue for researchers and practitioners in natural language understanding, though it is incremental as it identifies a problem rather than solving it.
The paper analyzed negation in eight popular NLU corpora and found they contain few negations compared to general English, often unimportant ones, and that ignoring negations still yields correct predictions. It showed state-of-the-art transformers perform substantially worse on instances with negation, especially important ones, concluding new corpora are needed to handle negation in NLU tasks.
This paper analyzes negation in eight popular corpora spanning six natural language understanding tasks. We show that these corpora have few negations compared to general-purpose English, and that the few negations in them are often unimportant. Indeed, one can often ignore negations and still make the right predictions. Additionally, experimental results show that state-of-the-art transformers trained with these corpora obtain substantially worse results with instances that contain negation, especially if the negations are important. We conclude that new corpora accounting for negation are needed to solve natural language understanding tasks when negation is present.