To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks
This provides practical guidelines for NLP practitioners on model adaptation, but it is incremental as it builds on existing pretraining methods.
The paper investigates whether to fine-tune or use feature extraction when adapting pretrained models to NLP tasks, finding that the best method depends on task similarity, with empirical results across diverse tasks using state-of-the-art models.
While most previous work has focused on different pretraining objectives and architectures for transfer learning, we ask how to best adapt the pretrained model to a given target task. We focus on the two most common forms of adaptation, feature extraction (where the pretrained weights are frozen), and directly fine-tuning the pretrained model. Our empirical results across diverse NLP tasks with two state-of-the-art models show that the relative performance of fine-tuning vs. feature extraction depends on the similarity of the pretraining and target tasks. We explore possible explanations for this finding and provide a set of adaptation guidelines for the NLP practitioner.