Automated Generation of Multilingual Clusters for the Evaluation of Distributed Representations
This work addresses the need for better intrinsic evaluation tools for word embeddings in NLP, though it appears incremental as it builds on existing outlier detection tasks.
The authors tackled the problem of evaluating word embeddings by proposing a language-agnostic method to automatically generate semantically similar clusters and outliers, creating the WikiSem500 dataset. They found that performance on this dataset correlates with sentiment analysis results, with specific numbers not provided in the abstract.
We propose a language-agnostic way of automatically generating sets of semantically similar clusters of entities along with sets of "outlier" elements, which may then be used to perform an intrinsic evaluation of word embeddings in the outlier detection task. We used our methodology to create a gold-standard dataset, which we call WikiSem500, and evaluated multiple state-of-the-art embeddings. The results show a correlation between performance on this dataset and performance on sentiment analysis.