Multilinguality Does not Make Sense: Investigating Factors Behind Zero-Shot Transfer in Sense-Aware Tasks
This challenges common assumptions in cross-lingual NLP, offering insights for low-resource languages, though it is incremental as it focuses on specific tasks.
The study investigated the assumption that training on more languages improves zero-shot transfer in sense-aware tasks like polysemy and lexical semantic change, finding that multilinguality is not necessary for effective transfer, with analysis across 28 languages showing other factors like data differences and evaluation artifacts better explain the benefits.
Cross-lingual transfer is central to modern NLP, enabling models to perform tasks in languages different from those they were trained on. A common assumption is that training on more languages improves zero-shot transfer. We test this on sense-aware tasks-polysemy and lexical semantic change-and find that multilinguality is not necessary for effective transfer. Our large-scale analysis across 28 languages reveals that other factors, such as differences in pretraining and fine-tuning data and evaluation artifacts, better explain the perceived benefits of multilinguality. We also release fine-tuned models and provide empirical baselines to support future research. While focused on two sense-aware tasks, our findings offer broader insights into cross-lingual transfer, especially for low-resource languages.