Deconstructing and reconstructing word embedding algorithms
This provides a foundational understanding for NLP researchers by unifying and improving word embedding methods, though it is incremental as it builds on existing algorithms.
The paper deconstructs word embedding algorithms like Word2vec and GloVe to identify key conditions for performance, revealing they approximate pointwise mutual information and balance loss gradients, and uses these insights to create Hilbert-MLE, which achieves equivalent or better performance on 17 datasets.
Uncontextualized word embeddings are reliable feature representations of words used to obtain high quality results for various NLP applications. Given the historical success of word embeddings in NLP, we propose a retrospective on some of the most well-known word embedding algorithms. In this work, we deconstruct Word2vec, GloVe, and others, into a common form, unveiling some of the necessary and sufficient conditions required for making performant word embeddings. We find that each algorithm: (1) fits vector-covector dot products to approximate pointwise mutual information (PMI); and, (2) modulates the loss gradient to balance weak and strong signals. We demonstrate that these two algorithmic features are sufficient conditions to construct a novel word embedding algorithm, Hilbert-MLE. We find that its embeddings obtain equivalent or better performance against other algorithms across 17 intrinsic and extrinsic datasets.