MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning
This addresses a challenging and under-studied problem for NLP in low-resource languages, offering a novel method to enhance transfer learning where traditional approaches struggle.
The paper tackles the problem of cross-lingual transfer learning for extremely low-resource languages lacking large-scale data, proposing MetaXL to transform representations from auxiliary languages to improve transfer, with experiments showing effectiveness on tasks like sentiment analysis and named entity recognition.
The combination of multilingual pre-trained representations and cross-lingual transfer learning is one of the most effective methods for building functional NLP systems for low-resource languages. However, for extremely low-resource languages without large-scale monolingual corpora for pre-training or sufficient annotated data for fine-tuning, transfer learning remains an under-studied and challenging task. Moreover, recent work shows that multilingual representations are surprisingly disjoint across languages, bringing additional challenges for transfer onto extremely low-resource languages. In this paper, we propose MetaXL, a meta-learning based framework that learns to transform representations judiciously from auxiliary languages to a target one and brings their representation spaces closer for effective transfer. Extensive experiments on real-world low-resource languages - without access to large-scale monolingual corpora or large amounts of labeled data - for tasks like cross-lingual sentiment analysis and named entity recognition show the effectiveness of our approach. Code for MetaXL is publicly available at github.com/microsoft/MetaXL.