Sentence Representation Learning with Generative Objective rather than Contrastive Objective
This addresses the need for more interpretable and effective sentence representations in NLP, offering a novel approach that improves over existing methods.
The paper tackles the problem of learning sentence-level representations by proposing a generative self-supervised objective based on phrase reconstruction, which outperforms state-of-the-art contrastive methods on STS benchmarks and downstream semantic tasks with performance improvements.
Though offering amazing contextualized token-level representations, current pre-trained language models take less attention on accurately acquiring sentence-level representation during their self-supervised pre-training. However, contrastive objectives which dominate the current sentence representation learning bring little linguistic interpretability and no performance guarantee on downstream semantic tasks. We instead propose a novel generative self-supervised learning objective based on phrase reconstruction. To overcome the drawbacks of previous generative methods, we carefully model intra-sentence structure by breaking down one sentence into pieces of important phrases. Empirical studies show that our generative learning achieves powerful enough performance improvement and outperforms the current state-of-the-art contrastive methods not only on the STS benchmarks, but also on downstream semantic retrieval and reranking tasks. Our code is available at https://github.com/chengzhipanpan/PaSeR.