Controlled Experiments for Word Embeddings
This work provides insights into word embedding properties for researchers in natural language processing, though it is incremental as it builds on existing models like word2vec.
The paper tackled the problem of understanding how word frequency and co-occurrence noise affect word embeddings by proposing controlled experiments with modified training corpora, revealing that word vector length depends linearly on these factors with coefficients varying by word.
An experimental approach to studying the properties of word embeddings is proposed. Controlled experiments, achieved through modifications of the training corpus, permit the demonstration of direct relations between word properties and word vector direction and length. The approach is demonstrated using the word2vec CBOW model with experiments that independently vary word frequency and word co-occurrence noise. The experiments reveal that word vector length depends more or less linearly on both word frequency and the level of noise in the co-occurrence distribution of the word. The coefficients of linearity depend upon the word. The special point in feature space, defined by the (artificial) word with pure noise in its co-occurrence distribution, is found to be small but non-zero.