Mixed Membership Word Embeddings for Computational Social Science
This addresses the need for interpretable and data-efficient word embeddings in computational social science, though it is an incremental advancement combining existing techniques.
The authors tackled the problem of word embeddings being unsuitable for computational social science due to big data requirements and lack of interpretability, by proposing a probabilistic model-based method that improves predictive language modeling by up to 63% in MRR over skip-gram and enhances interpretability.
Word embeddings improve the performance of NLP systems by revealing the hidden structural relationships between words. Despite their success in many applications, word embeddings have seen very little use in computational social science NLP tasks, presumably due to their reliance on big data, and to a lack of interpretability. I propose a probabilistic model-based word embedding method which can recover interpretable embeddings, without big data. The key insight is to leverage mixed membership modeling, in which global representations are shared, but individual entities (i.e. dictionary words) are free to use these representations to uniquely differing degrees. I show how to train the model using a combination of state-of-the-art training techniques for word embeddings and topic models. The experimental results show an improvement in predictive language modeling of up to 63% in MRR over the skip-gram, and demonstrate that the representations are beneficial for supervised learning. I illustrate the interpretability of the models with computational social science case studies on State of the Union addresses and NIPS articles.