RetroGAN: A Cyclic Post-Specialization System for Improving Out-of-Knowledge and Rare Word Representations
This addresses the limitation of retrofitting for out-of-knowledge and rare words in NLP, but it is incremental as it builds on existing retrofitting and GAN methods.
The paper tackled the problem of retrofitting word vectors for concepts not present in a knowledge base by proposing RetroGAN, a cyclic post-specialization system using GANs, and achieved state-of-the-art results on word-similarity benchmarks, including CARD-660.
Retrofitting is a technique used to move word vectors closer together or further apart in their space to reflect their relationships in a Knowledge Base (KB). However, retrofitting only works on concepts that are present in that KB. RetroGAN uses a pair of Generative Adversarial Networks (GANs) to learn a one-to-one mapping between concepts and their retrofitted counterparts. It applies that mapping (post-specializes) to handle concepts that do not appear in the original KB in a manner similar to how some natural language systems handle out-of-vocabulary entries. We test our system on three word-similarity benchmarks and a downstream sentence simplification task and achieve the state of the art (CARD-660). Altogether, our results demonstrate our system's effectiveness for out-of-knowledge and rare word generalization.