RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank
This addresses the need for more nuanced sentence similarity ranking in NLP applications, representing an incremental improvement over existing contrastive learning methods.
The paper tackles the problem of unsupervised sentence representation learning by introducing RankCSE, which incorporates ranking consistency and distillation into contrastive learning to capture fine-grained ranking information, achieving superior performance on semantic textual similarity and transfer tasks over state-of-the-art baselines.
Unsupervised sentence representation learning is one of the fundamental problems in natural language processing with various downstream applications. Recently, contrastive learning has been widely adopted which derives high-quality sentence representations by pulling similar semantics closer and pushing dissimilar ones away. However, these methods fail to capture the fine-grained ranking information among the sentences, where each sentence is only treated as either positive or negative. In many real-world scenarios, one needs to distinguish and rank the sentences based on their similarities to a query sentence, e.g., very relevant, moderate relevant, less relevant, irrelevant, etc. In this paper, we propose a novel approach, RankCSE, for unsupervised sentence representation learning, which incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework. In particular, we learn semantically discriminative sentence representations by simultaneously ensuring ranking consistency between two representations with different dropout masks, and distilling listwise ranking knowledge from the teacher. An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks. Experimental results demonstrate the superior performance of our approach over several state-of-the-art baselines.