Annotating Training Data for Conditional Semantic Textual Similarity Measurement using Large Language Models
This work addresses a data bottleneck for researchers in natural language processing working on conditional semantic similarity, though it is incremental as it builds on prior annotation efforts.
The paper tackles the lack of large and accurately annotated datasets for the Conditional Semantic Textual Similarity (C-STS) task by using Large Language Models to correct condition statements and similarity ratings in an existing dataset, resulting in a 5.4% statistically significant improvement in Spearman correlation when training a supervised model on the cleaned data.
Semantic similarity between two sentences depends on the aspects considered between those sentences. To study this phenomenon, Deshpande et al. (2023) proposed the Conditional Semantic Textual Similarity (C-STS) task and annotated a human-rated similarity dataset containing pairs of sentences compared under two different conditions. However, Tu et al. (2024) found various annotation issues in this dataset and showed that manually re-annotating a small portion of it leads to more accurate C-STS models. Despite these pioneering efforts, the lack of large and accurately annotated C-STS datasets remains a blocker for making progress on this task as evidenced by the subpar performance of the C-STS models. To address this training data need, we resort to Large Language Models (LLMs) to correct the condition statements and similarity ratings in the original dataset proposed by Deshpande et al. (2023). Our proposed method is able to re-annotate a large training dataset for the C-STS task with minimal manual effort. Importantly, by training a supervised C-STS model on our cleaned and re-annotated dataset, we achieve a 5.4% statistically significant improvement in Spearman correlation. The re-annotated dataset is available at https://LivNLP.github.io/CSTS-reannotation.