On Dimensional Linguistic Properties of the Word Embedding Space
This work addresses the understanding of word embedding properties for NLP researchers, revealing counterintuitive limitations in common post-processing methods.
The paper analyzes word embeddings via principal components, finding that variance explained does not correlate with syntactic information or downstream performance, and shows that variance-based post-processing harms sentence classification and machine translation tasks.
Word embeddings have become a staple of several natural language processing tasks, yet much remains to be understood about their properties. In this work, we analyze word embeddings in terms of their principal components and arrive at a number of novel and counterintuitive observations. In particular, we characterize the utility of variance explained by the principal components as a proxy for downstream performance. Furthermore, through syntactic probing of the principal embedding space, we show that the syntactic information captured by a principal component does not correlate with the amount of variance it explains. Consequently, we investigate the limitations of variance based embedding post-processing and demonstrate that such post-processing is counter-productive in sentence classification and machine translation tasks. Finally, we offer a few precautionary guidelines on applying variance based embedding post-processing and explain why non-isotropic geometry might be integral to word embedding performance.