HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference
This addresses a specific bias issue in NLI models, which is incremental as it builds on known biases with new debiasing approaches.
The paper tackled the problem of hypothesis-only bias in natural language inference models, where predictions can be made correctly by ignoring the premise, and explored adversarial examples and debiasing methods like down-sampling and adversarial training to mitigate this bias, showing competitive results with baselines including BERT and RoBERTa.
Many recent studies have shown that for models trained on datasets for natural language inference (NLI), it is possible to make correct predictions by merely looking at the hypothesis while completely ignoring the premise. In this work, we manage to derive adversarial examples in terms of the hypothesis-only bias and explore eligible ways to mitigate such bias. Specifically, we extract various phrases from the hypotheses (artificial patterns) in the training sets, and show that they have been strong indicators to the specific labels. We then figure out `hard' and `easy' instances from the original test sets whose labels are opposite to or consistent with those indications. We also set up baselines including both pretrained models (BERT, RoBERTa, XLNet) and competitive non-pretrained models (InferSent, DAM, ESIM). Apart from the benchmark and baselines, we also investigate two debiasing approaches which exploit the artificial pattern modeling to mitigate such hypothesis-only bias: down-sampling and adversarial training. We believe those methods can be treated as competitive baselines in NLI debiasing tasks.