Intrinsic analysis for dual word embedding space models
This research provides guidance for researchers and practitioners on selecting optimal hyper-parameters for dual word embedding models, particularly for those working with Word2Vec and GloVe, by identifying configurations that yield better performance across various linguistic tasks.
This paper investigates the optimal configuration of dual word embedding space models by comparing 84 different Word2Vec and GloVe models across semantic, association, and analogy tasks using 9 linguistic datasets. It found that non-default model configurations outperformed default settings for Word2Vec in 2 out of 3 tasks, and for GloVe in all 3 evaluation tasks.
Recent word embeddings techniques represent words in a continuous vector space, moving away from the atomic and sparse representations of the past. Each such technique can further create multiple varieties of embeddings based on different settings of hyper-parameters like embedding dimension size, context window size and training method. One additional variety appears when we especially consider the Dual embedding space techniques which generate not one but two-word embeddings as output. This gives rise to an interesting question - "is there one or a combination of the two word embeddings variety, which works better for a specific task?". This paper tries to answer this question by considering all of these variations. Herein, we compare two classical embedding methods belonging to two different methodologies - Word2Vec from window-based and Glove from count-based. For an extensive evaluation after considering all variations, a total of 84 different models were compared against semantic, association and analogy evaluations tasks which are made up of 9 open-source linguistics datasets. The final Word2vec reports showcase the preference of non-default model for 2 out of 3 tasks. In case of Glove, non-default models outperform in all 3 evaluation tasks.