Virtual Augmentation Supported Contrastive Learning of Sentence Representations
This work addresses a key bottleneck in NLP for researchers and practitioners by enabling more effective unsupervised learning without domain-specific augmentation rules.
The paper tackles the challenge of designing data augmentations for contrastive learning in NLP by proposing VaSCL, which uses nearest neighbors in representation space to generate virtual augmentations, achieving new state-of-the-art results in unsupervised sentence representation learning.
Despite profound successes, contrastive representation learning relies on carefully designed data augmentations using domain specific knowledge. This challenge is magnified in natural language processing where no general rules exist for data augmentation due to the discrete nature of natural language. We tackle this challenge by presenting a Virtual augmentation Supported Contrastive Learning of sentence representations (VaSCL). Originating from the interpretation that data augmentation essentially constructs the neighborhoods of each training instance, we in turn utilize the neighborhood to generate effective data augmentations. Leveraging the large training batch size of contrastive learning, we approximate the neighborhood of an instance via its K-nearest in-batch neighbors in the representation space. We then define an instance discrimination task regarding this neighborhood and generate the virtual augmentation in an adversarial training manner. We access the performance of VaSCL on a wide range of downstream tasks, and set a new state-of-the-art for unsupervised sentence representation learning.