LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
This addresses the computational inefficiency of multilingual models for deployment in resource-limited settings, but it is incremental as it builds on existing distillation techniques.
The paper tackled the problem of deploying large multilingual BERT models on resource-restricted devices by proposing LightMBERT, a distillation method that transfers cross-lingual generalization to a smaller student model, achieving results comparable to the teacher mBERT and significantly outperforming baselines.
The multilingual pre-trained language models (e.g, mBERT, XLM and XLM-R) have shown impressive performance on cross-lingual natural language understanding tasks. However, these models are computationally intensive and difficult to be deployed on resource-restricted devices. In this paper, we propose a simple yet effective distillation method (LightMBERT) for transferring the cross-lingual generalization ability of the multilingual BERT to a small student model. The experiment results empirically demonstrate the efficiency and effectiveness of LightMBERT, which is significantly better than the baselines and performs comparable to the teacher mBERT.