Evaluating Word Embedding Models: Methods and Experimental Results
This work provides a comparative analysis for researchers and practitioners in natural language processing, but it is incremental as it builds on existing evaluation methods without introducing new models.
The paper conducted an extensive evaluation of six word embedding models using intrinsic and extrinsic evaluators, showing that different evaluators focus on different aspects and some correlate better with natural language processing tasks.
Extensive evaluation on a large number of word embedding models for language processing applications is conducted in this work. First, we introduce popular word embedding models and discuss desired properties of word models and evaluation methods (or evaluators). Then, we categorize evaluators into intrinsic and extrinsic two types. Intrinsic evaluators test the quality of a representation independent of specific natural language processing tasks while extrinsic evaluators use word embeddings as input features to a downstream task and measure changes in performance metrics specific to that task. We report experimental results of intrinsic and extrinsic evaluators on six word embedding models. It is shown that different evaluators focus on different aspects of word models, and some are more correlated with natural language processing tasks. Finally, we adopt correlation analysis to study performance consistency of extrinsic and intrinsic evalutors.