Pcc-tuning: Breaking the Contrastive Learning Ceiling in Semantic Textual Similarity
This addresses a critical bottleneck in computational linguistics for improving sentence embedding models, offering a novel solution to break through a performance ceiling in STS benchmarks.
The paper tackled the stagnation in Semantic Textual Similarity (STS) performance, where existing methods plateaued at an average Spearman's correlation of around 86.5, and proposed Pcc-tuning, which uses Pearson's correlation coefficient as a loss function to achieve a new state-of-the-art, surpassing previous methods with minimal fine-grained annotated samples.
Semantic Textual Similarity (STS) constitutes a critical research direction in computational linguistics and serves as a key indicator of the encoding capabilities of embedding models. Driven by advances in pre-trained language models and contrastive learning, leading sentence representation methods have reached an average Spearman's correlation score of approximately 86 across seven STS benchmarks in SentEval. However, further progress has become increasingly marginal, with no existing method attaining an average score higher than 86.5 on these tasks. This paper conducts an in-depth analysis of this phenomenon and concludes that the upper limit for Spearman's correlation scores under contrastive learning is 87.5. To transcend this ceiling, we propose an innovative approach termed Pcc-tuning, which employs Pearson's correlation coefficient as a loss function to refine model performance beyond contrastive learning. Experimental results demonstrate that Pcc-tuning can markedly surpass previous state-of-the-art strategies with only a minimal amount of fine-grained annotated samples.