Measuring Cross-Lingual Transferability of Multilingual Transformers on Sentence Classification
This work addresses the challenge of understanding and optimizing cross-lingual transfer in multilingual AI models, which is incremental as it builds on existing methods to improve measurement.
The paper tackled the problem of measuring cross-lingual transferability in multilingual Transformers for sentence classification by proposing IGap, a metric that considers training error and works without end-task data, and experimental results showed it outperforms baselines in transferability measuring and ranking.
Recent studies have exhibited remarkable capabilities of pre-trained multilingual Transformers, especially cross-lingual transferability. However, current methods do not measure cross-lingual transferability well, hindering the understanding of multilingual Transformers. In this paper, we propose IGap, a cross-lingual transferability metric for multilingual Transformers on sentence classification tasks. IGap takes training error into consideration, and can also estimate transferability without end-task data. Experimental results show that IGap outperforms baseline metrics for transferability measuring and transfer direction ranking. Besides, we conduct extensive systematic experiments where we compare transferability among various multilingual Transformers, fine-tuning algorithms, and transfer directions. More importantly, our results reveal three findings about cross-lingual transfer, which helps us to better understand multilingual Transformers.