Making Language Models Robust Against Negation
This addresses a long-standing challenge in making language models more robust for tasks involving negation, though it appears incremental as it builds on existing models like BERT and RoBERTa.
The paper tackles the problem of language models struggling with negation in natural language understanding by proposing a self-supervised method using novel tasks like Next Sentence Polarity Prediction, resulting in improvements of 1.8% to 9.1% on benchmarks such as CondaQA.
Negation has been a long-standing challenge for language models. Previous studies have shown that they struggle with negation in many natural language understanding tasks. In this work, we propose a self-supervised method to make language models more robust against negation. We introduce a novel task, Next Sentence Polarity Prediction (NSPP), and a variation of the Next Sentence Prediction (NSP) task. We show that BERT and RoBERTa further pre-trained on our tasks outperform the off-the-shelf versions on nine negation-related benchmarks. Most notably, our pre-training tasks yield between 1.8% and 9.1% improvement on CondaQA, a large question-answering corpus requiring reasoning over negation.