SMoA: Sparse Mixture of Adapters to Mitigate Multiple Dataset Biases
This addresses the issue of poor generalization in NLP models due to biases, offering a more efficient solution than prior methods, though it is incremental as it builds on adapter tuning.
The paper tackles the problem of multiple dataset biases in NLP tasks, which harm generalization and robustness, by proposing SMoA, a method that effectively mitigates these biases and outperforms existing debiasing techniques and baselines in experiments on Natural Language Inference and Paraphrase Identification tasks.
Recent studies reveal that various biases exist in different NLP tasks, and over-reliance on biases results in models' poor generalization ability and low adversarial robustness. To mitigate datasets biases, previous works propose lots of debiasing techniques to tackle specific biases, which perform well on respective adversarial sets but fail to mitigate other biases. In this paper, we propose a new debiasing method Sparse Mixture-of-Adapters (SMoA), which can mitigate multiple dataset biases effectively and efficiently. Experiments on Natural Language Inference and Paraphrase Identification tasks demonstrate that SMoA outperforms full-finetuning, adapter tuning baselines, and prior strong debiasing methods. Further analysis indicates the interpretability of SMoA that sub-adapter can capture specific pattern from the training data and specialize to handle specific bias.