Incremental Skip-gram Model with Negative Sampling
This addresses the need for incremental updates in word embeddings for applications requiring real-time learning, but it is incremental as it extends an existing method.
The paper tackled the problem that existing neural word embedding methods like skip-gram with negative sampling cannot update models incrementally, and presented a simple incremental extension with theoretical and empirical validation showing its practical usefulness.
This paper explores an incremental training strategy for the skip-gram model with negative sampling (SGNS) from both empirical and theoretical perspectives. Existing methods of neural word embeddings, including SGNS, are multi-pass algorithms and thus cannot perform incremental model update. To address this problem, we present a simple incremental extension of SGNS and provide a thorough theoretical analysis to demonstrate its validity. Empirical experiments demonstrated the correctness of the theoretical analysis as well as the practical usefulness of the incremental algorithm.