Explicit Word Density Estimation for Language Modelling
This work addresses limitations in language modeling for NLP applications, but appears incremental as it builds on existing NeuralODE and Normalizing Flow concepts.
The authors tackled the problem of language modeling by proposing a new family of models based on NeuralODEs and continuous Normalizing Flows, which improved on some baselines.
Language Modelling has been a central part of Natural Language Processing for a very long time and in the past few years LSTM-based language models have been the go-to method for commercial language modeling. Recently, it has been shown that when looking at language modelling from a matrix factorization point of view, the final Softmax layer limits the expressiveness of the model, by putting an upper bound on the rank of the resulting matrix. Additionally, a new family of neural networks based called NeuralODEs, has been introduced as a continuous alternative to Residual Networks. Moreover, it has been shown that there is a connection between these models and Normalizing Flows. In this work we propose a new family of language models based on NeuralODEs and the continuous analogue of Normalizing Flows and manage to improve on some of the baselines.