Adversarial Contrastive Estimation
This work addresses the challenge of enhancing representation learning in NLP and related domains through more effective negative sampling, though it is incremental as it builds on existing contrastive learning frameworks.
The paper tackled the problem of improving contrastive learning methods by augmenting the negative sampler with an adversarially learned component to find harder negative examples, resulting in faster convergence and improved metrics across tasks like word embeddings, order embeddings, and knowledge graph embeddings.
Learning by contrasting positive and negative samples is a general strategy adopted by many methods. Noise contrastive estimation (NCE) for word embeddings and translating embeddings for knowledge graphs are examples in NLP employing this approach. In this work, we view contrastive learning as an abstraction of all such methods and augment the negative sampler into a mixture distribution containing an adversarially learned sampler. The resulting adaptive sampler finds harder negative examples, which forces the main model to learn a better representation of the data. We evaluate our proposal on learning word embeddings, order embeddings and knowledge graph embeddings and observe both faster convergence and improved results on multiple metrics.