Multi-Level Contrastive Learning for Cross-Lingual Alignment
This work addresses the problem of enhancing cross-lingual transfer for NLP applications, offering incremental improvements over existing methods.
The paper tackled improving cross-lingual alignment in pre-trained models by proposing a multi-level contrastive learning framework that integrates word-level information and uses a novel loss to handle training issues, resulting in significant performance gains on zero-shot cross-lingual tasks in the Xtreme benchmark.
Cross-language pre-trained models such as multilingual BERT (mBERT) have achieved significant performance in various cross-lingual downstream NLP tasks. This paper proposes a multi-level contrastive learning (ML-CTL) framework to further improve the cross-lingual ability of pre-trained models. The proposed method uses translated parallel data to encourage the model to generate similar semantic embeddings for different languages. However, unlike the sentence-level alignment used in most previous studies, in this paper, we explicitly integrate the word-level information of each pair of parallel sentences into contrastive learning. Moreover, cross-zero noise contrastive estimation (CZ-NCE) loss is proposed to alleviate the impact of the floating-point error in the training process with a small batch size. The proposed method significantly improves the cross-lingual transfer ability of our basic model (mBERT) and outperforms on multiple zero-shot cross-lingual downstream tasks compared to the same-size models in the Xtreme benchmark.