How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks
This work addresses the evaluation of word embeddings for NLP researchers, offering a more complete performance analysis, though it is incremental as it refines existing evaluation practices rather than introducing a new method.
The paper tackles the problem of evaluating word embeddings by arguing that current methods fail to assess data efficiency and accessible information, proposing a shift towards evaluating simple supervised tasks with varied data amounts. It demonstrates this approach through a comprehensive evaluation, revealing that word similarity and analogy information is non-linearly encoded, questioning unsupervised cosine-based methods.
Maybe the single most important goal of representation learning is making subsequent learning faster. Surprisingly, this fact is not well reflected in the way embeddings are evaluated. In addition, recent practice in word embeddings points towards importance of learning specialized representations. We argue that focus of word representation evaluation should reflect those trends and shift towards evaluating what useful information is easily accessible. Specifically, we propose that evaluation should focus on data efficiency and simple supervised tasks, where the amount of available data is varied and scores of a supervised model are reported for each subset (as commonly done in transfer learning). In order to illustrate significance of such analysis, a comprehensive evaluation of selected word embeddings is presented. Proposed approach yields a more complete picture and brings new insight into performance characteristics, for instance information about word similarity or analogy tends to be non--linearly encoded in the embedding space, which questions the cosine-based, unsupervised, evaluation methods. All results and analysis scripts are available online.