Explicit Alignment Objectives for Multilingual Bidirectional Encoders
This work addresses the challenge of improving cross-lingual transfer for NLP tasks, particularly benefiting low-resource languages, though it is incremental as it builds on existing encoder frameworks.
The paper tackles the problem of aligning multilingual embeddings in pre-trained encoders by introducing AMBER, a method with explicit alignment objectives, which achieves gains of up to 1.1 average F1 score on sequence tagging and 27.3 average accuracy on retrieval compared to XLMR-large.
Pre-trained cross-lingual encoders such as mBERT (Devlin et al., 2019) and XLMR (Conneau et al., 2020) have proven to be impressively effective at enabling transfer-learning of NLP systems from high-resource languages to low-resource languages. This success comes despite the fact that there is no explicit objective to align the contextual embeddings of words/sentences with similar meanings across languages together in the same space. In this paper, we present a new method for learning multilingual encoders, AMBER (Aligned Multilingual Bidirectional EncodeR). AMBER is trained on additional parallel data using two explicit alignment objectives that align the multilingual representations at different granularities. We conduct experiments on zero-shot cross-lingual transfer learning for different tasks including sequence tagging, sentence retrieval and sentence classification. Experimental results show that AMBER obtains gains of up to 1.1 average F1 score on sequence tagging and up to 27.3 average accuracy on retrieval over the XLMR-large model which has 3.2x the parameters of AMBER. Our code and models are available at http://github.com/junjiehu/amber.