Self-Adaptive Reconstruction with Contrastive Learning for Unsupervised Sentence Embeddings
This addresses the issue of fine-grained semantic loss in sentence embeddings for NLP applications, though it appears incremental as it builds on existing contrastive learning methods.
The paper tackles the problem of token bias in pretrained language models for unsupervised sentence embeddings, which leads to poor semantic capture. The proposed SARCSE framework achieves significant improvements over the strong baseline SimCSE on 7 STS tasks.
Unsupervised sentence embeddings task aims to convert sentences to semantic vector representations. Most previous works directly use the sentence representations derived from pretrained language models. However, due to the token bias in pretrained language models, the models can not capture the fine-grained semantics in sentences, which leads to poor predictions. To address this issue, we propose a novel Self-Adaptive Reconstruction Contrastive Sentence Embeddings (SARCSE) framework, which reconstructs all tokens in sentences with an AutoEncoder to help the model to preserve more fine-grained semantics during tokens aggregating. In addition, we proposed a self-adaptive reconstruction loss to alleviate the token bias towards frequency. Experimental results show that SARCSE gains significant improvements compared with the strong baseline SimCSE on the 7 STS tasks.