Non-contrastive sentence representations via self-supervision
This work addresses the challenge of improving unsupervised sentence representations for NLP applications, but it is incremental as it builds on existing contrastive and dimension contrastive methods.
The paper tackled the problem of learning unsupervised sentence embeddings by comparing dimension contrastive methods with the standard SimCSE baseline, finding that dimension contrastive objectives can outperform SimCSE on downstream tasks without auxiliary losses.
Sample contrastive methods, typically referred to simply as contrastive are the foundation of most unsupervised methods to learn text and sentence embeddings. On the other hand, a different class of self-supervised loss functions and methods have been considered in the computer vision community and referred to as dimension contrastive. In this paper, we thoroughly compare this class of methods with the standard baseline for contrastive sentence embeddings, SimCSE. We find that self-supervised embeddings trained using dimension contrastive objectives can outperform SimCSE on downstream tasks without needing auxiliary loss functions.