Efficient Fine-Tuning of Compressed Language Models with Learners
This addresses computational challenges in fine-tuning compressed language models for NLP practitioners, though it is incremental as it builds on existing compression and fine-tuning methods.
The paper tackles the problem of resource-intensive fine-tuning of BERT-based models by introducing Learner modules and priming, which fine-tune a subset of parameters to achieve faster convergence and lower resource usage. Results show learners train 7x fewer parameters on GLUE and fine-tune 20% faster on CoLA while performing on par with or surpassing baselines.
Fine-tuning BERT-based models is resource-intensive in memory, computation, and time. While many prior works aim to improve inference efficiency via compression techniques, e.g., pruning, these works do not explicitly address the computational challenges of training to downstream tasks. We introduce Learner modules and priming, novel methods for fine-tuning that exploit the overparameterization of pre-trained language models to gain benefits in convergence speed and resource utilization. Learner modules navigate the double bind of 1) training efficiently by fine-tuning a subset of parameters, and 2) training effectively by ensuring quick convergence and high metric scores. Our results on DistilBERT demonstrate that learners perform on par with or surpass the baselines. Learners train 7x fewer parameters than state-of-the-art methods on GLUE. On CoLA, learners fine-tune 20% faster, and have significantly lower resource utilization.