Identification, Interpretability, and Bayesian Word Embeddings
This work addresses the challenge for social scientists in integrating word embeddings into regression frameworks, offering a method to improve interpretability and usability in text analysis.
The paper tackles the problem of using word embeddings in social science regression by introducing Bayesian Word Embeddings with Automatic Relevance Determination priors and anchoring methods to make embeddings identified and interpretable. It applies this model to analyze inaugural addresses and foreign policy deliberations, finding that internationalist rhetoric decreased after 1945 and that bellicosity correlates with hostile actions.
Social scientists have recently turned to analyzing text using tools from natural language processing like word embeddings to measure concepts like ideology, bias, and affinity. However, word embeddings are difficult to use in the regression framework familiar to social scientists: embeddings are are neither identified, nor directly interpretable. I offer two advances on standard embedding models to remedy these problems. First, I develop Bayesian Word Embeddings with Automatic Relevance Determination priors, relaxing the assumption that all embedding dimensions have equal weight. Second, I apply work identifying latent variable models to anchor the dimensions of the resulting embeddings, identifying them, and making them interpretable and usable in a regression. I then apply this model and anchoring approach to two cases, the shift in internationalist rhetoric in the American presidents' inaugural addresses, and the relationship between bellicosity in American foreign policy decision-makers' deliberations. I find that inaugural addresses became less internationalist after 1945, which goes against the conventional wisdom, and that an increase in bellicosity is associated with an increase in hostile actions by the United States, showing that elite deliberations are not cheap talk, and helping confirm the validity of the model.