ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
This addresses the issue of poor sentence representations for natural language processing tasks, offering an incremental improvement in unsupervised fine-tuning for BERT models.
The paper tackled the problem of collapsed sentence representations from BERT models in semantic textual similarity tasks by proposing ConSERT, a contrastive self-supervised framework, achieving an 8% relative improvement over previous state-of-the-art and comparable performance to supervised methods with only 1000 samples.
Learning high-quality sentence representations benefits a wide range of natural language processing tasks. Though BERT-based pre-trained language models achieve high performance on many downstream tasks, the native derived sentence representations are proved to be collapsed and thus produce a poor performance on the semantic textual similarity (STS) tasks. In this paper, we present ConSERT, a Contrastive Framework for Self-Supervised Sentence Representation Transfer, that adopts contrastive learning to fine-tune BERT in an unsupervised and effective way. By making use of unlabeled texts, ConSERT solves the collapse issue of BERT-derived sentence representations and make them more applicable for downstream tasks. Experiments on STS datasets demonstrate that ConSERT achieves an 8\% relative improvement over the previous state-of-the-art, even comparable to the supervised SBERT-NLI. And when further incorporating NLI supervision, we achieve new state-of-the-art performance on STS tasks. Moreover, ConSERT obtains comparable results with only 1000 samples available, showing its robustness in data scarcity scenarios.