Can Eye Movement Data Be Used As Ground Truth For Word Embeddings Evaluation?
This work addresses the challenge of finding robust evaluation methods for word embeddings, which is crucial for NLP researchers, but it appears to be an incremental study with limited impact.
The paper investigates whether eye movement data from silent reading can serve as a reliable ground truth for evaluating distributional semantic models, testing this hypothesis across English and Russian datasets. The results indicate that the hypothesis may not hold, as correlations between embeddings and eye movement data were insufficient to support its validity.
In recent years a certain success in the task of modeling lexical semantics was obtained with distributional semantic models. Nevertheless, the scientific community is still unaware what is the most reliable evaluation method for these models. Some researchers argue that the only possible gold standard could be obtained from neuro-cognitive resources that store information about human cognition. One of such resources is eye movement data on silent reading. The goal of this work is to test the hypothesis of whether such data could be used to evaluate distributional semantic models on different languages. We propose experiments with English and Russian eye movement datasets (Provo Corpus, GECO and Russian Sentence Corpus), word vectors (Skip-Gram models trained on national corpora and Web corpora) and word similarity datasets of Russian and English assessed by humans in order to find the existence of correlation between embeddings and eye movement data and test the hypothesis that this correlation is language independent. As a result, we found that the validity of the hypothesis being tested could be questioned.