Lookup-Table Recurrent Language Models for Long Tail Speech Recognition
This addresses the challenge of improving long tail performance in speech recognition for applications requiring efficient large-scale language models, though it is an incremental advancement over existing methods.
The paper tackles the problem of scaling RNN language models efficiently by introducing LookupLM, which uses an n-gram embedding table to increase expressivity without extra operations, resulting in a 2.44 improvement in long tail log perplexity and a 23.4% reduction in WER on a speech recognition task.
We introduce Lookup-Table Language Models (LookupLM), a method for scaling up the size of RNN language models with only a constant increase in the floating point operations, by increasing the expressivity of the embedding table. In particular, we instantiate an (additional) embedding table which embeds the previous n-gram token sequence, rather than a single token. This allows the embedding table to be scaled up arbitrarily -- with a commensurate increase in performance -- without changing the token vocabulary. Since embeddings are sparsely retrieved from the table via a lookup; increasing the size of the table adds neither extra operations to each forward pass nor extra parameters that need to be stored on limited GPU/TPU memory. We explore scaling n-gram embedding tables up to nearly a billion parameters. When trained on a 3-billion sentence corpus, we find that LookupLM improves long tail log perplexity by 2.44 and long tail WER by 23.4% on a downstream speech recognition task over a standard RNN language model baseline, an improvement comparable to a scaling up the baseline by 6.2x the number of floating point operations.