Dynamic Corrective Self-Distillation for Better Fine-Tuning of Pretrained Models
This addresses performance degradation during fine-tuning for NLP practitioners, but it is incremental as it builds on existing self-distillation and adaptive boosting ideas.
The paper tackles the problem of aggressive fine-tuning in pre-trained language models with limited labeled data, which causes performance decline, by proposing a dynamic corrective self-distillation approach that improves fine-tuning capability, leading to enhanced performance and robustness as demonstrated on the GLUE benchmark.
We tackle the challenging issue of aggressive fine-tuning encountered during the process of transfer learning of pre-trained language models (PLMs) with limited labeled downstream data. This problem primarily results in a decline in performance on the subsequent task. Inspired by the adaptive boosting method in traditional machine learning, we present an effective dynamic corrective self-distillation (DCS) approach to improve the fine-tuning of the PLMs. Our technique involves performing a self-distillation mechanism where, at each iteration, the student model actively adapts and corrects itself by dynamically adjusting the weights assigned to individual data points. This iterative self-correcting process significantly enhances the overall fine-tuning capability of PLMs, leading to improved performance and robustness. We conducted comprehensive evaluations using the GLUE benchmark demonstrating the efficacy of our method in enhancing the fine-tuning process for various PLMs across diverse downstream tasks.