Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings
This work addresses the challenge of cross-lingual transfer for NLP applications, but it is incremental as it builds on existing alignment methods.
The paper tackled the problem of aligning multilingual contextualized embeddings by proposing Optimal Transport as an alignment objective during fine-tuning, achieving improvements over baselines and competitive results on XNLI and XQuAD tasks.
Recent studies have proposed different methods to improve multilingual word representations in contextualized settings including techniques that align between source and target embedding spaces. For contextualized embeddings, alignment becomes more complex as we additionally take context into consideration. In this work, we propose using Optimal Transport (OT) as an alignment objective during fine-tuning to further improve multilingual contextualized representations for downstream cross-lingual transfer. This approach does not require word-alignment pairs prior to fine-tuning that may lead to sub-optimal matching and instead learns the word alignments within context in an unsupervised manner. It also allows different types of mappings due to soft matching between source and target sentences. We benchmark our proposed method on two tasks (XNLI and XQuAD) and achieve improvements over baselines as well as competitive results compared to similar recent works.