Lightweight Adaptive Mixture of Neural and N-gram Language Models
This work addresses the challenge of optimizing ensemble methods for language modeling, which is incremental as it builds on existing ensemble techniques.
The paper tackles the problem of improving ensemble performance in language modeling by proposing a method to adaptively combine neural and n-gram models at each time step, resulting in a significant improvement over the state-of-the-art ensemble on the One Billion Word benchmark without retraining the base models.
It is often the case that the best performing language model is an ensemble of a neural language model with n-grams. In this work, we propose a method to improve how these two models are combined. By using a small network which predicts the mixture weight between the two models, we adapt their relative importance at each time step. Because the gating network is small, it trains quickly on small amounts of held out data, and does not add overhead at scoring time. Our experiments carried out on the One Billion Word benchmark show a significant improvement over the state of the art ensemble without retraining of the basic modules.