Low-Rank RNN Adaptation for Context-Aware Language Modeling
This work addresses the need for more effective context-aware language modeling in applications like personalized or domain-specific text prediction, representing an incremental improvement over existing embedding-based methods.
The paper tackled the problem of adapting RNN-based language models to context information by proposing a low-rank transformation mechanism for the recurrent layer weight matrix, resulting in improved perplexity and classification performance across various context types.
A context-aware language model uses location, user and/or domain metadata (context) to adapt its predictions. In neural language models, context information is typically represented as an embedding and it is given to the RNN as an additional input, which has been shown to be useful in many applications. We introduce a more powerful mechanism for using context to adapt an RNN by letting the context vector control a low-rank transformation of the recurrent layer weight matrix. Experiments show that allowing a greater fraction of the model parameters to be adjusted has benefits in terms of perplexity and classification for several different types of context.