Cross-lingual Transferring of Pre-trained Contextualized Language Models
This addresses the challenge of developing NLP models for languages with limited corpora, offering a more efficient alternative to full pre-training, though it is incremental based on existing cross-lingual and translation methods.
The paper tackles the problem of training pre-trained contextualized language models (PrLMs) for non-English languages by proposing TreLM, a cross-lingual transferring framework that uses an intermediate TRILayer and adversarial embedding alignment to handle language differences. Experiments show it outperforms training from scratch with limited data in performance and efficiency, and is more economical even with resource-rich data.
Though the pre-trained contextualized language model (PrLM) has made a significant impact on NLP, training PrLMs in languages other than English can be impractical for two reasons: other languages often lack corpora sufficient for training powerful PrLMs, and because of the commonalities among human languages, computationally expensive PrLM training for different languages is somewhat redundant. In this work, building upon the recent works connecting cross-lingual model transferring and neural machine translation, we thus propose a novel cross-lingual model transferring framework for PrLMs: TreLM. To handle the symbol order and sequence length differences between languages, we propose an intermediate ``TRILayer" structure that learns from these differences and creates a better transfer in our primary translation direction, as well as a new cross-lingual language modeling objective for transfer training. Additionally, we showcase an embedding aligning that adversarially adapts a PrLM's non-contextualized embedding space and the TRILayer structure to learn a text transformation network across languages, which addresses the vocabulary difference between languages. Experiments on both language understanding and structure parsing tasks show the proposed framework significantly outperforms language models trained from scratch with limited data in both performance and efficiency. Moreover, despite an insignificant performance loss compared to pre-training from scratch in resource-rich scenarios, our cross-lingual model transferring framework is significantly more economical.