Mitigating Shortcuts in Language Models with Soft Label Encoding
This work addresses the issue of spurious correlations in language models for natural language understanding tasks, offering a method to enhance robustness, though it appears incremental as it builds on existing debiasing techniques.
The paper tackles the problem of language models relying on spurious correlations in data by proposing Soft Label Encoding (SoftLE), a debiasing framework that modifies ground truth labels to reduce shortcuts, resulting in significant improvements in out-of-distribution generalization while maintaining in-distribution accuracy.
Recent research has shown that large language models rely on spurious correlations in the data for natural language understanding (NLU) tasks. In this work, we aim to answer the following research question: Can we reduce spurious correlations by modifying the ground truth labels of the training data? Specifically, we propose a simple yet effective debiasing framework, named Soft Label Encoding (SoftLE). We first train a teacher model with hard labels to determine each sample's degree of relying on shortcuts. We then add one dummy class to encode the shortcut degree, which is used to smooth other dimensions in the ground truth label to generate soft labels. This new ground truth label is used to train a more robust student model. Extensive experiments on two NLU benchmark tasks demonstrate that SoftLE significantly improves out-of-distribution generalization while maintaining satisfactory in-distribution accuracy.