Model Selection for Cross-Lingual Transfer
This addresses a practical issue in multilingual NLP for researchers and practitioners by improving model selection without target-language data, though it is incremental as it builds on existing transfer methods.
The paper tackles the problem of selecting the best fine-tuned model for cross-lingual transfer in zero-shot settings, where no target-language data is available, by proposing a method that uses small amounts of auxiliary pivot-language data and the model's internal representations, achieving results comparable to using target-language data across 25 languages.
Transformers that are pre-trained on multilingual corpora, such as, mBERT and XLM-RoBERTa, have achieved impressive cross-lingual transfer capabilities. In the zero-shot transfer setting, only English training data is used, and the fine-tuned model is evaluated on another target language. While this works surprisingly well, substantial variance has been observed in target language performance between different fine-tuning runs, and in the zero-shot setup, no target-language development data is available to select among multiple fine-tuned models. Prior work has relied on English dev data to select among models that are fine-tuned with different learning rates, number of steps and other hyperparameters, often resulting in suboptimal choices. In this paper, we show that it is possible to select consistently better models when small amounts of annotated data are available in auxiliary pivot languages. We propose a machine learning approach to model selection that uses the fine-tuned model's own internal representations to predict its cross-lingual capabilities. In extensive experiments we find that this method consistently selects better models than English validation data across twenty five languages (including eight low-resource languages), and often achieves results that are comparable to model selection using target language development data.