Model-Based Ranking of Source Languages for Zero-Shot Cross-Lingual Transfer
This work addresses the challenge of selecting optimal source languages for cross-lingual NLP tasks, which is incremental as it builds on existing multilingual models and ranking methods.
The paper tackles the problem of ranking source languages for zero-shot cross-lingual transfer by introducing NN-Rank, an algorithm that uses hidden representations from multilingual models and unlabeled target-language data, achieving average improvements of up to 35.56 NDCG for POS and 18.14 NDCG for NER compared to state-of-the-art baselines.
We present NN-Rank, an algorithm for ranking source languages for cross-lingual transfer, which leverages hidden representations from multilingual models and unlabeled target-language data. We experiment with two pretrained multilingual models and two tasks: part-of-speech tagging (POS) and named entity recognition (NER). We consider 51 source languages and evaluate on 56 and 72 target languages for POS and NER, respectively. When using in-domain data, NN-Rank beats state-of-the-art baselines that leverage lexical and linguistic features, with average improvements of up to 35.56 NDCG for POS and 18.14 NDCG for NER. As prior approaches can fall back to language-level features if target language data is not available, we show that NN-Rank remains competitive using only the Bible, an out-of-domain corpus available for a large number of languages. Ablations on the amount of unlabeled target data show that, for subsets consisting of as few as 25 examples, NN-Rank produces high-quality rankings which achieve 92.8% of the NDCG achieved using all available target data for ranking.