Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models
This work addresses the problem of understanding cross-domain transfer for researchers and practitioners, but it is incremental as it builds on existing evidence to explore specific conditions.
The study investigated the limits of cross-domain knowledge transfer for pretrained models by testing transfer into scrambled domains where word identity is removed, finding that BERT shows high transfer for classification but not sequence labeling tasks. Results quantify transfer rates and analyze contributions of pretraining, fine-tuning, and word frequency.
There is growing evidence that pretrained language models improve task-specific fine-tuning not just for the languages seen in pretraining, but also for new languages and even non-linguistic data. What is the nature of this surprising cross-domain transfer? We offer a partial answer via a systematic exploration of how much transfer occurs when models are denied any information about word identity via random scrambling. In four classification tasks and two sequence labeling tasks, we evaluate baseline models, LSTMs using GloVe embeddings, and BERT. We find that only BERT shows high rates of transfer into our scrambled domains, and for classification but not sequence labeling tasks. Our analyses seek to explain why transfer succeeds for some tasks but not others, to isolate the separate contributions of pretraining versus fine-tuning, and to quantify the role of word frequency. These findings help explain where and why cross-domain transfer occurs, which can guide future studies and practical fine-tuning efforts.