Word Embeddings: Stability and Semantic Change
This addresses reliability issues in NLP for researchers and practitioners, though it is incremental as it builds on existing embedding methods.
The paper investigates the instability of word embedding techniques like word2vec, GloVe, and fastText, showing that repeated training on the same data can yield different results, and proposes a method to reduce this instability by averaging multiple runs, which is applied to improve semantic change detection.
Word embeddings are computed by a class of techniques within natural language processing (NLP), that create continuous vector representations of words in a language from a large text corpus. The stochastic nature of the training process of most embedding techniques can lead to surprisingly strong instability, i.e. subsequently applying the same technique to the same data twice, can produce entirely different results. In this work, we present an experimental study on the instability of the training process of three of the most influential embedding techniques of the last decade: word2vec, GloVe and fastText. Based on the experimental results, we propose a statistical model to describe the instability of embedding techniques and introduce a novel metric to measure the instability of the representation of an individual word. Finally, we propose a method to minimize the instability - by computing a modified average over multiple runs - and apply it to a specific linguistic problem: The detection and quantification of semantic change, i.e. measuring changes in the meaning and usage of words over time.