PATS: Sensitivity-aware Noisy Learning for Pretrained Language Models
This addresses the issue of inefficient fine-tuning for NLP practitioners, though it is incremental as it builds on existing fine-tuning methods.
The paper tackles the problem of redundant parameters in fine-tuned pretrained language models, which leads to suboptimal performance, by introducing PATS, a noisy training mechanism that adds noise based on parameter sensitivity, resulting in consistent performance improvements across GLUE benchmark tasks.
A wide range of NLP tasks benefit from the fine-tuning of pretrained language models (PLMs). However, a number of redundant parameters which contribute less to the downstream task are observed in a directly fine-tuned model. We consider the gap between pretraining and downstream tasks hinders the training of these redundant parameters, and results in a suboptimal performance of the overall model. In this paper, we present PATS (Perturbation According To Sensitivity), a noisy training mechanism which considers each parameter's importance in the downstream task to help fine-tune PLMs. The main idea of PATS is to add bigger noise to parameters with lower sensitivity and vice versa, in order to activate more parameters' contributions to downstream tasks without affecting the sensitive ones much. Extensive experiments conducted on different tasks of the GLUE benchmark show PATS can consistently empower the fine-tuning of different sizes of PLMs, and the parameters in the well-performing models always have more concentrated distributions of sensitivities, which experimentally proves the effectiveness of our method.