UNSEE: Unsupervised Non-contrastive Sentence Embeddings
This work addresses a specific challenge in NLP for researchers and practitioners, representing an incremental improvement in unsupervised sentence embedding methods.
The paper tackled the problem of representation collapse in unsupervised sentence embeddings by proposing a target network to enable non-contrastive objectives, achieving peak performance in the Massive Text Embedding benchmark and outperforming SimCSE.
We present UNSEE: Unsupervised Non-Contrastive Sentence Embeddings, a novel approach that outperforms SimCSE in the Massive Text Embedding benchmark. Our exploration begins by addressing the challenge of representation collapse, a phenomenon observed when contrastive objectives in SimCSE are replaced with non-contrastive objectives. To counter this issue, we propose a straightforward solution known as the target network, effectively mitigating representation collapse. The introduction of the target network allows us to leverage non-contrastive objectives, maintaining training stability while achieving performance improvements comparable to contrastive objectives. Our method has achieved peak performance in non-contrastive sentence embeddings through meticulous fine-tuning and optimization. This comprehensive effort has yielded superior sentence representation models, showcasing the effectiveness of our approach.