WOVe: Incorporating Word Order in GloVe Word Embeddings
This addresses a limitation in word embedding models for natural language processing applications, but it is incremental as it builds directly on the existing GloVe framework.
The paper tackled the problem of GloVe word embeddings not explicitly considering word order within contexts, and proposed multiple methods to incorporate word order, resulting in a 36.34% improvement in accuracy on a word analogy task compared to the baseline.
Word vector representations open up new opportunities to extract useful information from unstructured text. Defining a word as a vector made it easy for the machine learning algorithms to understand a text and extract information from. Word vector representations have been used in many applications such word synonyms, word analogy, syntactic parsing, and many others. GloVe, based on word contexts and matrix vectorization, is an ef-fective vector-learning algorithm. It improves on previous vector-learning algorithms. However, the GloVe model fails to explicitly consider the order in which words appear within their contexts. In this paper, multiple methods of incorporating word order in GloVe word embeddings are proposed. Experimental results show that our Word Order Vector (WOVe) word embeddings approach outperforms unmodified GloVe on the natural lan-guage tasks of analogy completion and word similarity. WOVe with direct concatenation slightly outperformed GloVe on the word similarity task, increasing average rank by 2%. However, it greatly improved on the GloVe baseline on a word analogy task, achieving an average 36.34% improvement in accuracy.