Denoising Word Embeddings by Averaging in a Shared Space
This work addresses noise and instability in word embeddings for NLP applications, offering an incremental improvement over existing fusion methods.
The paper tackles the problem of improving word embedding quality by fusing multiple embeddings trained on the same corpus with different initializations, projecting them into a shared space using Generalized Procrustes Analysis, resulting in consistent improvements over raw models and simple averaging across tasks, with noticeable gains in rare word evaluations.
We introduce a new approach for smoothing and improving the quality of word embeddings. We consider a method of fusing word embeddings that were trained on the same corpus but with different initializations. We project all the models to a shared vector space using an efficient implementation of the Generalized Procrustes Analysis (GPA) procedure, previously used in multilingual word translation. Our word representation demonstrates consistent improvements over the raw models as well as their simplistic average, on a range of tasks. As the new representations are more stable and reliable, there is a noticeable improvement in rare word evaluations.