DBR: Divergence-Based Regularization for Debiasing Natural Language Understanding Models
This addresses the issue of shortcut learning in NLU models for improving generalization, but it is incremental as it builds on existing regularization techniques.
The paper tackled the problem of pre-trained language models relying on superficial features and shortcuts in natural language understanding tasks, which hinders generalization to out-of-domain data, and proposed Divergence-Based Regularization (DBR) to mitigate this, resulting in improved out-of-domain performance with minimal in-domain accuracy loss across three NLU tasks.
Pre-trained language models (PLMs) have achieved impressive results on various natural language processing tasks. However, recent research has revealed that these models often rely on superficial features and shortcuts instead of developing a genuine understanding of language, especially for natural language understanding (NLU) tasks. Consequently, the models struggle to generalize to out-of-domain data. In this work, we propose Divergence Based Regularization (DBR) to mitigate this shortcut learning behavior. Our method measures the divergence between the output distributions for original examples and examples where shortcut tokens have been masked. This process prevents the model's predictions from being overly influenced by shortcut features or biases. We evaluate our model on three NLU tasks and find that it improves out-of-domain performance with little loss of in-domain accuracy. Our results demonstrate that reducing the reliance on shortcuts and superficial features can enhance the generalization ability of large pre-trained language models.