Information-Weighted Neural Cache Language Models for ASR
This work addresses incremental improvements in language modeling for ASR systems, potentially enhancing accuracy for speech recognition users.
The paper tackled improving neural cache language models for automatic speech recognition by introducing information-weighted interpolation and selective caching of content words, achieving a 29.9%/32.1% relative perplexity improvement on WikiText-2 and significant WER reductions on WSJ.
Neural cache language models (LMs) extend the idea of regular cache language models by making the cache probability dependent on the similarity between the current context and the context of the words in the cache. We make an extensive comparison of 'regular' cache models with neural cache models, both in terms of perplexity and WER after rescoring first-pass ASR results. Furthermore, we propose two extensions to this neural cache model that make use of the content value/information weight of the word: firstly, combining the cache probability and LM probability with an information-weighted interpolation and secondly, selectively adding only content words to the cache. We obtain a 29.9%/32.1% (validation/test set) relative improvement in perplexity with respect to a baseline LSTM LM on the WikiText-2 dataset, outperforming previous work on neural cache LMs. Additionally, we observe significant WER reductions with respect to the baseline model on the WSJ ASR task.