Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model
This addresses cross-lingual natural language understanding, particularly for low-resource languages, but is incremental as it builds on existing teacher-student frameworks.
The paper tackles the problem of improving multilingual language models by learning semantic structure, proposing Multi-level Multilingual Knowledge Distillation (MMKD) to align multiple levels between teacher and student models, resulting in outperforming baselines on XNLI and XQuAD and gains on low-resource languages.
Pre-trained multilingual language models play an important role in cross-lingual natural language understanding tasks. However, existing methods did not focus on learning the semantic structure of representation, and thus could not optimize their performance. In this paper, we propose Multi-level Multilingual Knowledge Distillation (MMKD), a novel method for improving multilingual language models. Specifically, we employ a teacher-student framework to adopt rich semantic representation knowledge in English BERT. We propose token-, word-, sentence-, and structure-level alignment objectives to encourage multiple levels of consistency between source-target pairs and correlation similarity between teacher and student models. We conduct experiments on cross-lingual evaluation benchmarks including XNLI, PAWS-X, and XQuAD. Experimental results show that MMKD outperforms other baseline models of similar size on XNLI and XQuAD and obtains comparable performance on PAWS-X. Especially, MMKD obtains significant performance gains on low-resource languages.