A Rank-Based Similarity Metric for Word Embeddings
This work addresses the need for better evaluation metrics in NLP for researchers and practitioners, but it is incremental as it builds on existing similarity tasks.
The paper tackles the problem of evaluating word embeddings by proposing a rank-based similarity metric, which performs comparably to vector cosine in similarity estimation and outperforms it in outlier detection, suggesting improved clustering quality.
Word Embeddings have recently imposed themselves as a standard for representing word meaning in NLP. Semantic similarity between word pairs has become the most common evaluation benchmark for these representations, with vector cosine being typically used as the only similarity metric. In this paper, we report experiments with a rank-based metric for WE, which performs comparably to vector cosine in similarity estimation and outperforms it in the recently-introduced and challenging task of outlier detection, thus suggesting that rank-based measures can improve clustering quality.