Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging
This work addresses the need for efficient generalization in NLP tasks, offering a simple optimization alternative to knowledge distillation, though it is incremental as it adapts an existing method.
The authors tackled the problem of improving generalization for compact pre-trained language models by adapting Stochastic Weight Averaging (SWA) to fine-tuning, which outperformed state-of-the-art knowledge distillation methods without extra computational cost.
Knowledge Distillation (KD) is a commonly used technique for improving the generalization of compact Pre-trained Language Models (PLMs) on downstream tasks. However, such methods impose the additional burden of training a separate teacher model for every new dataset. Alternatively, one may directly work on the improvement of the optimization procedure of the compact model toward better generalization. Recent works observe that the flatness of the local minimum correlates well with better generalization. In this work, we adapt Stochastic Weight Averaging (SWA), a method encouraging convergence to a flatter minimum, to fine-tuning PLMs. We conduct extensive experiments on various NLP tasks (text classification, question answering, and generation) and different model architectures and demonstrate that our adaptation improves the generalization without extra computation cost. Moreover, we observe that this simple optimization technique is able to outperform the state-of-the-art KD methods for compact models.