Neural Random Projections for Language Modelling
This addresses the problem of parameter efficiency and unknown vocabulary handling in language modeling for NLP researchers, though it is incremental as it builds on existing neural and random projection techniques.
The paper tackles data sparsity in neural language models by encoding words with fixed sparse random representations and projecting them to a smaller embedding space, resulting in more compact models with reduced parameters while maintaining performance on the Penn Treebank corpus.
Neural network-based language models deal with data sparsity problems by mapping the large discrete space of words into a smaller continuous space of real-valued vectors. By learning distributed vector representations for words, each training sample informs the neural network model about a combinatorial number of other patterns. In this paper, we exploit the sparsity in natural language even further by encoding each unique input word using a fixed sparse random representation. These sparse codes are then projected onto a smaller embedding space which allows for the encoding of word occurrences from a possibly unknown vocabulary, along with the creation of more compact language models using a reduced number of parameters. We investigate the properties of our encoding mechanism empirically, by evaluating its performance on the widely used Penn Treebank corpus. We show that guaranteeing approximately equidistant (nearly orthogonal) vector representations for unique discrete inputs is enough to provide the neural network model with enough information to learn --and make use-- of distributed representations for these inputs.