Self-Normalized Importance Sampling for Neural Language Modeling
This work addresses the training efficiency problem for large vocabulary neural language models, particularly in speech recognition, but is incremental as it builds on prior sampling-based methods.
The authors tackled the computational bottleneck of softmax normalization in neural language models by proposing self-normalized importance sampling, which eliminates the need for a correction step and achieves competitive performance in automatic speech recognition tasks.
To mitigate the problem of having to traverse over the full vocabulary in the softmax normalization of a neural language model, sampling-based training criteria are proposed and investigated in the context of large vocabulary word-based neural language models. These training criteria typically enjoy the benefit of faster training and testing, at a cost of slightly degraded performance in terms of perplexity and almost no visible drop in word error rate. While noise contrastive estimation is one of the most popular choices, recently we show that other sampling-based criteria can also perform well, as long as an extra correction step is done, where the intended class posterior probability is recovered from the raw model outputs. In this work, we propose self-normalized importance sampling. Compared to our previous work, the criteria considered in this work are self-normalized and there is no need to further conduct a correction step. Through self-normalized language model training as well as lattice rescoring experiments, we show that our proposed self-normalized importance sampling is competitive in both research-oriented and production-oriented automatic speech recognition tasks.