Generative or Contrastive? Phrase Reconstruction for Better Sentence Representation Learning
This addresses the challenge of sentence representation learning for NLP practitioners, offering an incremental improvement by integrating generative and contrastive approaches.
The paper tackles the problem of acquiring sentence-level representations from pre-trained language models by proposing a generative self-supervised learning objective based on phrase reconstruction. The result shows that this method achieves performance on par with contrastive learning in STS tasks and outperforms state-of-the-art SimCSE in unsupervised semantic retrieval tasks.
Though offering amazing contextualized token-level representations, current pre-trained language models actually take less attention on acquiring sentence-level representation during its self-supervised pre-training. If self-supervised learning can be distinguished into two subcategories, generative and contrastive, then most existing studies show that sentence representation learning may more benefit from the contrastive methods but not the generative methods. However, contrastive learning cannot be well compatible with the common token-level generative self-supervised learning, and does not guarantee good performance on downstream semantic retrieval tasks. Thus, to alleviate such obvious inconveniences, we instead propose a novel generative self-supervised learning objective based on phrase reconstruction. Empirical studies show that our generative learning may yield powerful enough sentence representation and achieve performance in Sentence Textual Similarity (STS) tasks on par with contrastive learning. Further, in terms of unsupervised setting, our generative method outperforms previous state-of-the-art SimCSE on the benchmark of downstream semantic retrieval tasks.