Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studies
This work addresses the challenge of understanding transfer learning bottlenecks for researchers designing language transfer scenarios, though it is incremental in nature.
The paper investigates factors that make cross-lingual transfer learning difficult by systematically altering aspects like syntax and vocabulary in the GLUE benchmark, finding that models struggle with vocabulary misalignment and embedding re-initialization, with performance drops persisting even after 15 million tokens of continued pretraining.
When we transfer a pretrained language model to a new language, there are many axes of variation that change at once. To disentangle the impact of different factors like syntactic similarity and vocabulary similarity, we propose a set of controlled transfer studies: we systematically transform the language of the GLUE benchmark, altering one axis of crosslingual variation at a time, and then measure the resulting drops in a pretrained model's downstream performance. We find that models can largely recover from syntactic-style shifts, but cannot recover from vocabulary misalignment and embedding matrix re-initialization, even with continued pretraining on 15 million tokens. %On the other hand, transferring to a dataset with an unaligned vocabulary is extremely hard to recover from in the low-data regime. Moreover, good-quality tokenizers in the transfer language do not make vocabulary alignment easier. Our experiments provide insights into the factors of cross-lingual transfer that researchers should most focus on when designing language transfer scenarios.