EMS: Efficient and Effective Massively Multilingual Sentence Embedding Learning
This work addresses the inefficiency in training multilingual models for researchers and practitioners, offering a more resource-effective solution, though it is incremental as it builds on existing methods like LASER and LaBSE.
The paper tackles the problem of high computational cost in training massively multilingual sentence embedding models by introducing EMS, which uses cross-lingual token-level reconstruction and sentence-level contrastive learning to achieve comparable or better results on tasks like cross-lingual sentence retrieval with significantly fewer parallel sentences and GPU resources.
Massively multilingual sentence representation models, e.g., LASER, SBERT-distill, and LaBSE, help significantly improve cross-lingual downstream tasks. However, the use of a large amount of data or inefficient model architectures results in heavy computation to train a new model according to our preferred languages and domains. To resolve this issue, we introduce efficient and effective massively multilingual sentence embedding (EMS), using cross-lingual token-level reconstruction (XTR) and sentence-level contrastive learning as training objectives. Compared with related studies, the proposed model can be efficiently trained using significantly fewer parallel sentences and GPU computation resources. Empirical results showed that the proposed model significantly yields better or comparable results with regard to cross-lingual sentence retrieval, zero-shot cross-lingual genre classification, and sentiment classification. Ablative analyses demonstrated the efficiency and effectiveness of each component of the proposed model. We release the codes for model training and the EMS pre-trained sentence embedding model, which supports 62 languages ( https://github.com/Mao-KU/EMS ).