Character-Level Language Modeling with Hierarchical Recurrent Neural Networks
This work addresses the problem of inefficient character-level language modeling for natural language processing tasks, offering a novel architecture that improves performance while reducing parameters, though it is incremental in advancing existing RNN methods.
The authors tackled the performance gap between character-level and word-level language models by proposing hierarchical RNN architectures with multiple timescales, achieving better perplexity than Kneser-Ney 5-gram word-level models on the One Billion Word Benchmark with only 2% of parameters and improving recognition accuracies on the WSJ corpus with a 30% parameter reduction.
Recurrent neural network (RNN) based character-level language models (CLMs) are extremely useful for modeling out-of-vocabulary words by nature. However, their performance is generally much worse than the word-level language models (WLMs), since CLMs need to consider longer history of tokens to properly predict the next one. We address this problem by proposing hierarchical RNN architectures, which consist of multiple modules with different timescales. Despite the multi-timescale structures, the input and output layers operate with the character-level clock, which allows the existing RNN CLM training approaches to be directly applicable without any modifications. Our CLM models show better perplexity than Kneser-Ney (KN) 5-gram WLMs on the One Billion Word Benchmark with only 2% of parameters. Also, we present real-time character-level end-to-end speech recognition examples on the Wall Street Journal (WSJ) corpus, where replacing traditional mono-clock RNN CLMs with the proposed models results in better recognition accuracies even though the number of parameters are reduced to 30%.