Incremental Adaptation Strategies for Neural Network Language Models
This work addresses the problem of slow model adaptation for neural network language models in applications like speech recognition and machine translation, but it is incremental as it builds on existing adaptation techniques.
The paper tackles the slow training of neural network language models by proposing two efficient adaptation methods, continued training on resampled data and insertion of adaptation layers, which achieve significant improvements without overfitting in a CAT environment.
It is today acknowledged that neural network language models outperform backoff language models in applications like speech recognition or statistical machine translation. However, training these models on large amounts of data can take several days. We present efficient techniques to adapt a neural network language model to new data. Instead of training a completely new model or relying on mixture approaches, we propose two new methods: continued training on resampled data or insertion of adaptation layers. We present experimental results in an CAT environment where the post-edits of professional translators are used to improve an SMT system. Both methods are very fast and achieve significant improvements without overfitting the small adaptation data.