Intrinsic Subspace Evaluation of Word Embedding Representations
This provides a more nuanced evaluation framework for NLP researchers working with word embeddings, though it appears incremental as it builds on existing intrinsic evaluation concepts.
The authors tackled the problem of evaluating word embedding representations by introducing a new methodology that tests whether representations contain subspaces necessary to satisfy four fundamental natural language criteria, demonstrating the limits of existing point-based evaluations and applying it to compare count vector and neural network models.
We introduce a new methodology for intrinsic evaluation of word representations. Specifically, we identify four fundamental criteria based on the characteristics of natural language that pose difficulties to NLP systems; and develop tests that directly show whether or not representations contain the subspaces necessary to satisfy these criteria. Current intrinsic evaluations are mostly based on the overall similarity or full-space similarity of words and thus view vector representations as points. We show the limits of these point-based intrinsic evaluations. We apply our evaluation methodology to the comparison of a count vector model and several neural network models and demonstrate important properties of these models.