An Analysis on the Learning Rules of the Skip-Gram Model
This work addresses a theoretical gap for researchers in natural language processing, but it is incremental as it builds on the widely used word2vec model.
The paper tackles the lack of understanding of the skip-ram model's mechanism by deriving its learning rules and linking them to competitive learning, and it provides and validates global optimal solution constraints through experiments.
To improve the generalization of the representations for natural language processing tasks, words are commonly represented using vectors, where distances among the vectors are related to the similarity of the words. While word2vec, the state-of-the-art implementation of the skip-gram model, is widely used and improves the performance of many natural language processing tasks, its mechanism is not yet well understood. In this work, we derive the learning rules for the skip-gram model and establish their close relationship to competitive learning. In addition, we provide the global optimal solution constraints for the skip-gram model and validate them by experimental results.