Meta-Learning a Dynamical Language Model
This work addresses language modeling efficiency by integrating dynamic weight updates, but it appears incremental as it builds on prior experiments with evolving weights.
The authors tackled word-level language modeling by combining short-term hidden states with medium-term representations in dynamically evolving weights, using a meta-learner trained via gradient descent to continuously update the model weights.
We consider the task of word-level language modeling and study the possibility of combining hidden-states-based short-term representations with medium-term representations encoded in dynamical weights of a language model. Our work extends recent experiments on language models with dynamically evolving weights by casting the language modeling problem into an online learning-to-learn framework in which a meta-learner is trained by gradient-descent to continuously update a language model weights.