Advancing Semantic Textual Similarity Modeling: A Regression Framework with Translated ReLU and Smooth K2 Loss
This work addresses challenges for researchers in STS tasks involving nuanced similarity levels or limited computational resources, though it appears incremental as it builds on existing methods like BERT and RoBERTa.
The paper tackles the problem of Semantic Textual Similarity (STS) modeling by addressing limitations in contrastive learning and Sentence-BERT, which fail to leverage fine-grained similarity levels or handle progressive semantic relationships, and proposes a regression framework with Translated ReLU and Smooth K2 Loss functions, achieving convincing performance across seven established STS benchmarks.
Since the introduction of BERT and RoBERTa, research on Semantic Textual Similarity (STS) has made groundbreaking progress. Particularly, the adoption of contrastive learning has substantially elevated state-of-the-art performance across various STS benchmarks. However, contrastive learning categorizes text pairs as either semantically similar or dissimilar, failing to leverage fine-grained annotated information and necessitating large batch sizes to prevent model collapse. These constraints pose challenges for researchers engaged in STS tasks that involve nuanced similarity levels or those with limited computational resources, compelling them to explore alternatives like Sentence-BERT. Despite its efficiency, Sentence-BERT tackles STS tasks from a classification perspective, overlooking the progressive nature of semantic relationships, which results in suboptimal performance. To bridge this gap, this paper presents an innovative regression framework and proposes two simple yet effective loss functions: Translated ReLU and Smooth K2 Loss. Experimental results demonstrate that our method achieves convincing performance across seven established STS benchmarks and offers the potential for further optimization of contrastive learning pre-trained models.