A Simple Language Model based on PMI Matrix Approximations
This work presents an incremental improvement for natural language processing researchers by offering a simpler method for language modeling based on existing techniques.
The authors tackled the problem of learning language models by training them to estimate word-context pointwise mutual information (PMI) and deriving conditional probabilities from it, showing that minor modifications to word2vec's algorithm yield principled models related to Noise Contrastive Estimation (NCE) based language models.
In this study, we introduce a new approach for learning language models by training them to estimate word-context pointwise mutual information (PMI), and then deriving the desired conditional probabilities from PMI at test time. Specifically, we show that with minor modifications to word2vec's algorithm, we get principled language models that are closely related to the well-established Noise Contrastive Estimation (NCE) based language models. A compelling aspect of our approach is that our models are trained with the same simple negative sampling objective function that is commonly used in word2vec to learn word embeddings.