Delta Embedding Learning
This addresses the limitation of semantics in unsupervised embeddings for NLP practitioners, but it is incremental as it builds on existing embedding fine-tuning methods.
The paper tackles the problem of suboptimal performance from inadequate fine-tuning of unsupervised word embeddings in NLP tasks by proposing Delta Embedding Learning, which uses structured regularization to incrementally tune embeddings, resulting in consistent performance improvements across various tasks and better semantic properties.
Unsupervised word embeddings have become a popular approach of word representation in NLP tasks. However there are limitations to the semantics represented by unsupervised embeddings, and inadequate fine-tuning of embeddings can lead to suboptimal performance. We propose a novel learning technique called Delta Embedding Learning, which can be applied to general NLP tasks to improve performance by optimized tuning of the word embeddings. A structured regularization is applied to the embeddings to ensure they are tuned in an incremental way. As a result, the tuned word embeddings become better word representations by absorbing semantic information from supervision without "forgetting." We apply the method to various NLP tasks and see a consistent improvement in performance. Evaluation also confirms the tuned word embeddings have better semantic properties.