Online Representation Learning in Recurrent Neural Language Models
This work addresses efficiency and accuracy challenges in language modeling for NLP applications, but it appears incremental as it extends existing online learning methods.
The authors tackled the problem of improving recurrent neural network language models by introducing continuous online learning with adaptive vector representations, resulting in increased language modeling accuracy and reduced parameter storage and computational requirements.
We investigate an extension of continuous online learning in recurrent neural network language models. The model keeps a separate vector representation of the current unit of text being processed and adaptively adjusts it after each prediction. The initial experiments give promising results, indicating that the method is able to increase language modelling accuracy, while also decreasing the parameters needed to store the model along with the computation required at each step.