Sparse Overcomplete Word Vector Representations
This work addresses the interpretability gap in word representations for NLP practitioners, though it is incremental as it builds on existing vector methods.
The paper tackled the problem of dense, uninterpretable word vectors by proposing methods to transform them into sparse, interpretable vectors, resulting in improved performance on benchmark tasks.
Current distributed representations of words show little resemblance to theories of lexical semantics. The former are dense and uninterpretable, the latter largely based on familiar, discrete classes (e.g., supersenses) and relations (e.g., synonymy and hypernymy). We propose methods that transform word vectors into sparse (and optionally binary) vectors. The resulting representations are more similar to the interpretable features typically used in NLP, though they are discovered automatically from raw corpora. Because the vectors are highly sparse, they are computationally easy to work with. Most importantly, we find that they outperform the original vectors on benchmark tasks.