Issues in evaluating semantic spaces using word analogies
This addresses a methodological issue in NLP for researchers evaluating word embeddings, but it is incremental as it critiques an existing tool without introducing a new paradigm.
The paper identifies a flaw in the offset method for evaluating semantic vector spaces, showing that it conflates offset consistency with neighborhood structure, and proposes simple baselines to improve evaluation utility.
The offset method for solving word analogies has become a standard evaluation tool for vector-space semantic models: it is considered desirable for a space to represent semantic relations as consistent vector offsets. We show that the method's reliance on cosine similarity conflates offset consistency with largely irrelevant neighborhood structure, and propose simple baselines that should be used to improve the utility of the method in vector space evaluation.