UniBERT: Adversarial Training for Language-Universal Representations
This work addresses the need for scalable and robust multilingual NLP models, offering incremental improvements in cross-lingual generalization for researchers and practitioners.
The paper tackled the problem of building efficient multilingual language models by introducing UniBERT, which integrates masked language modeling, adversarial training, and knowledge distillation, resulting in an average relative improvement of 7.72% over baselines across tasks like named entity recognition and question answering.
This paper presents UniBERT, a compact multilingual language model that uses an innovative training framework that integrates three components: masked language modeling, adversarial training, and knowledge distillation. Pre-trained on a meticulously curated Wikipedia corpus spanning 107 languages, UniBERT is designed to reduce the computational demands of large-scale models while maintaining competitive performance across various natural language processing tasks. Comprehensive evaluations on four tasks - named entity recognition, natural language inference, question answering, and semantic textual similarity - demonstrate that our multilingual training strategy enhanced by an adversarial objective significantly improves cross-lingual generalization. Specifically, UniBERT models show an average relative improvement of 7.72% over traditional baselines, which achieved an average relative improvement of only 1.17%, and statistical analysis confirms the significance of these gains (p-value = 0.0181). This work highlights the benefits of combining adversarial training and knowledge distillation to build scalable and robust language models, thus advancing the field of multilingual and cross-lingual natural language processing.