HiCL: Hierarchical Contrastive Learning of Unsupervised Sentence Embeddings
This work addresses the challenge of generalizing sentence embeddings to shorter texts for NLP applications, representing an incremental improvement over existing methods.
The paper tackles the problem of improving unsupervised sentence embeddings by proposing HiCL, a hierarchical contrastive learning framework that models both local segment-level and global sequence-level relationships, resulting in enhanced performance over the prior top-performing SNCSE model with average increases of +0.2% on BERT-large and +0.44% on RoBERTa-large across seven STS tasks.
In this paper, we propose a hierarchical contrastive learning framework, HiCL, which considers local segment-level and global sequence-level relationships to improve training efficiency and effectiveness. Traditional methods typically encode a sequence in its entirety for contrast with others, often neglecting local representation learning, leading to challenges in generalizing to shorter texts. Conversely, HiCL improves its effectiveness by dividing the sequence into several segments and employing both local and global contrastive learning to model segment-level and sequence-level relationships. Further, considering the quadratic time complexity of transformers over input tokens, HiCL boosts training efficiency by first encoding short segments and then aggregating them to obtain the sequence representation. Extensive experiments show that HiCL enhances the prior top-performing SNCSE model across seven extensively evaluated STS tasks, with an average increase of +0.2% observed on BERT-large and +0.44% on RoBERTa-large.