Universal Sentence Representation Learning with Conditional Masked Language Model
This work addresses the problem of learning high-quality, unsupervised universal sentence representations for researchers and developers working with natural language processing, offering significant gains in cross-lingual tasks.
This paper introduces Conditional Masked Language Modeling (CMLM) to learn sentence representations from unlabeled text. The English CMLM model achieves state-of-the-art performance on SentEval, and a multilingual CMLM model co-trained with bitext retrieval and NLI tasks improves cross-lingual semantic search by 10% over baselines.
This paper presents a novel training method, Conditional Masked Language Modeling (CMLM), to effectively learn sentence representations on large scale unlabeled corpora. CMLM integrates sentence representation learning into MLM training by conditioning on the encoded vectors of adjacent sentences. Our English CMLM model achieves state-of-the-art performance on SentEval, even outperforming models learned using supervised signals. As a fully unsupervised learning method, CMLM can be conveniently extended to a broad range of languages and domains. We find that a multilingual CMLM model co-trained with bitext retrieval (BR) and natural language inference (NLI) tasks outperforms the previous state-of-the-art multilingual models by a large margin, e.g. 10% improvement upon baseline models on cross-lingual semantic search. We explore the same language bias of the learned representations, and propose a simple, post-training and model agnostic approach to remove the language identifying information from the representation while still retaining sentence semantics.