Boosting the Performance of Transformer Architectures for Semantic Textual Similarity
This is an incremental improvement attempt for researchers working on semantic textual similarity tasks.
The authors fine-tuned transformer architectures (BERT, RoBERTa, DeBERTaV3) for semantic textual similarity on the STS Benchmark, combining model outputs with handmade features in boosting algorithms, but encountered worse test set results despite validation improvements, leading them to experiment with dataset splits and provide error analysis.
Semantic textual similarity is the task of estimating the similarity between the meaning of two texts. In this paper, we fine-tune transformer architectures for semantic textual similarity on the Semantic Textual Similarity Benchmark by tuning the model partially and then end-to-end. We experiment with BERT, RoBERTa, and DeBERTaV3 cross-encoders by approaching the problem as a binary classification task or a regression task. We combine the outputs of the transformer models and use handmade features as inputs for boosting algorithms. Due to worse test set results coupled with improvements on the validation set, we experiment with different dataset splits to further investigate this occurrence. We also provide an error analysis, focused on the edges of the prediction range.