Future Word Contexts in Neural Network Language Models
This addresses computational challenges in speech recognition for researchers and practitioners, though it is incremental as it builds on existing bi-RNNLM improvements.
The paper tackled the inefficiency of bidirectional recurrent neural network language models (bi-RNNLMs) by proposing a novel succeeding word RNNLM (su-RNNLM) that uses a feedforward unit to model a finite number of future words, resulting in more efficient training and lattice rescoring with only slight degradation compared to bi-RNNLMs on a meeting transcription task.
Recently, bidirectional recurrent network language models (bi-RNNLMs) have been shown to outperform standard, unidirectional, recurrent neural network language models (uni-RNNLMs) on a range of speech recognition tasks. This indicates that future word context information beyond the word history can be useful. However, bi-RNNLMs pose a number of challenges as they make use of the complete previous and future word context information. This impacts both training efficiency and their use within a lattice rescoring framework. In this paper these issues are addressed by proposing a novel neural network structure, succeeding word RNNLMs (su-RNNLMs). Instead of using a recurrent unit to capture the complete future word contexts, a feedforward unit is used to model a finite number of succeeding, future, words. This model can be trained much more efficiently than bi-RNNLMs and can also be used for lattice rescoring. Experimental results on a meeting transcription task (AMI) show the proposed model consistently outperformed uni-RNNLMs and yield only a slight degradation compared to bi-RNNLMs in N-best rescoring. Additionally, performance improvements can be obtained using lattice rescoring and subsequent confusion network decoding.