Learning Word Representations with Hierarchical Sparse Coding
This addresses the problem of scalable and effective word representation learning for natural language processing tasks, offering incremental improvements in speed and performance.
The paper tackles learning word representations by proposing a hierarchical sparse coding method with an efficient algorithm, achieving faster training on billions of tokens and outperforming or matching state-of-the-art results on benchmarks like word similarity and sentiment analysis.
We propose a new method for learning word representations using hierarchical regularization in sparse coding inspired by the linguistic study of word meanings. We show an efficient learning algorithm based on stochastic proximal methods that is significantly faster than previous approaches, making it possible to perform hierarchical sparse coding on a corpus of billions of word tokens. Experiments on various benchmark tasks---word similarity ranking, analogies, sentence completion, and sentiment analysis---demonstrate that the method outperforms or is competitive with state-of-the-art methods. Our word representations are available at \url{http://www.ark.cs.cmu.edu/dyogatam/wordvecs/}.