Mutual Information Maximization for Simple and Accurate Part-Of-Speech Induction
This work addresses part-of-speech induction for natural language processing, presenting an incremental improvement with a simple method.
The paper tackles part-of-speech induction by maximizing mutual information between induced labels and context, using two SGD-friendly objectives, and finds that a variational lower bound is robust while a generalized Brown objective is vulnerable, achieving competitive performance across multiple datasets and languages with a simple architecture.
We address part-of-speech (POS) induction by maximizing the mutual information between the induced label and its context. We focus on two training objectives that are amenable to stochastic gradient descent (SGD): a novel generalization of the classical Brown clustering objective and a recently proposed variational lower bound. While both objectives are subject to noise in gradient updates, we show through analysis and experiments that the variational lower bound is robust whereas the generalized Brown objective is vulnerable. We obtain competitive performance on a multitude of datasets and languages with a simple architecture that encodes morphology and context.