How transfer learning impacts linguistic knowledge in deep NLP models?
This work addresses the impact of transfer learning on linguistic knowledge in NLP models, providing insights for researchers and practitioners, but it is incremental as it builds on existing diagnostic methods.
The study investigated how fine-tuning pre-trained language models (BERT, RoBERTa, XLNet) on downstream NLP tasks affects their learned linguistic knowledge, finding that linguistic information becomes localized to lower layers after fine-tuning, with variations across architectures, such as BERT retaining it deeper than others.
Transfer learning from pre-trained neural language models towards downstream tasks has been a predominant theme in NLP recently. Several researchers have shown that deep NLP models learn non-trivial amount of linguistic knowledge, captured at different layers of the model. We investigate how fine-tuning towards downstream NLP tasks impacts the learned linguistic knowledge. We carry out a study across popular pre-trained models BERT, RoBERTa and XLNet using layer and neuron-level diagnostic classifiers. We found that for some GLUE tasks, the network relies on the core linguistic information and preserve it deeper in the network, while for others it forgets. Linguistic information is distributed in the pre-trained language models but becomes localized to the lower layers post fine-tuning, reserving higher layers for the task specific knowledge. The pattern varies across architectures, with BERT retaining linguistic information relatively deeper in the network compared to RoBERTa and XLNet, where it is predominantly delegated to the lower layers.