Streaming Word Embeddings with the Space-Saving Algorithm
This work addresses the need for efficient, low-memory word embeddings in real-time applications like social media analysis, though it is incremental as it adapts an existing method to a streaming context.
The paper tackled the problem of learning word embeddings in a streaming setting with bounded memory, proposing a streaming algorithm based on word2vec's skip-gram with negative sampling. The result was partial validation through experiments showing comparable cosine similarity and hashtag prediction performance on Twitter data.
We develop a streaming (one-pass, bounded-memory) word embedding algorithm based on the canonical skip-gram with negative sampling algorithm implemented in word2vec. We compare our streaming algorithm to word2vec empirically by measuring the cosine similarity between word pairs under each algorithm and by applying each algorithm in the downstream task of hashtag prediction on a two-month interval of the Twitter sample stream. We then discuss the results of these experiments, concluding they provide partial validation of our approach as a streaming replacement for word2vec. Finally, we discuss potential failure modes and suggest directions for future work.