Understanding and Improving Multi-Sense Word Embeddings via Extended Robust Principal Component Analysis
This addresses the issue of overly sensitive word embeddings for NLP researchers, but it is incremental as it builds on existing dimensionality reduction techniques.
The paper tackled the problem of pseudo multi-sense detection in unsupervised word embeddings by proposing Ex-RPCA, a novel principal analysis method, and showed that it improves embeddings by 5.6 points on the SCWS dataset.
Unsupervised learned representations of polysemous words generate a large of pseudo multi senses since unsupervised methods are overly sensitive to contextual variations. In this paper, we address the pseudo multi-sense detection for word embeddings by dimensionality reduction of sense pairs. We propose a novel principal analysis method, termed Ex-RPCA, designed to detect both pseudo multi senses and real multi senses. With Ex-RPCA, we empirically show that pseudo multi senses are generated systematically in unsupervised method. Moreover, the multi-sense word embeddings can by improved by a simple linear transformation based on Ex-RPCA. Our improved word embedding outperform the original one by 5.6 points on Stanford contextual word similarity (SCWS) dataset. We hope our simple yet effective approach will help the linguistic analysis of multi-sense word embeddings in the future.