On the Dimensionality of Word Embedding
This work addresses a foundational problem in natural language processing for researchers and practitioners by offering theoretical insights into word embedding dimensionality, though it is incremental as it builds on existing embedding techniques.
The paper tackles the problem of selecting the optimal dimensionality for word embeddings by proposing the Pairwise Inner Product (PIP) loss as a novel metric and revealing a fundamental bias-variance trade-off, which explains empirical observations like the existence of an optimal dimensionality and provides explicit guidance for dimensionality selection.
In this paper, we provide a theoretical understanding of word embedding and its dimensionality. Motivated by the unitary-invariance of word embedding, we propose the Pairwise Inner Product (PIP) loss, a novel metric on the dissimilarity between word embeddings. Using techniques from matrix perturbation theory, we reveal a fundamental bias-variance trade-off in dimensionality selection for word embeddings. This bias-variance trade-off sheds light on many empirical observations which were previously unexplained, for example the existence of an optimal dimensionality. Moreover, new insights and discoveries, like when and how word embeddings are robust to over-fitting, are revealed. By optimizing over the bias-variance trade-off of the PIP loss, we can explicitly answer the open question of dimensionality selection for word embedding.