Unsupervised hard Negative Augmentation for contrastive learning
This method addresses the challenge of creating effective negative samples for contrastive learning in NLP, particularly for semantic textual similarity, but appears incremental as it builds on existing augmentation techniques.
The paper tackles the problem of generating synthetic negative instances for contrastive learning by introducing Unsupervised hard Negative Augmentation (UNA), which uses TF-IDF scores to replace terms in sentences, resulting in improved performance in semantic textual similarity tasks, with additional gains when combined with paraphrasing augmentation.
We present Unsupervised hard Negative Augmentation (UNA), a method that generates synthetic negative instances based on the term frequency-inverse document frequency (TF-IDF) retrieval model. UNA uses TF-IDF scores to ascertain the perceived importance of terms in a sentence and then produces negative samples by replacing terms with respect to that. Our experiments demonstrate that models trained with UNA improve the overall performance in semantic textual similarity tasks. Additional performance gains are obtained when combining UNA with the paraphrasing augmentation. Further results show that our method is compatible with different backbone models. Ablation studies also support the choice of having a TF-IDF-driven control on negative augmentation.