Topically Driven Neural Language Model
This work addresses the limitation of context in language modeling for NLP applications, though it is incremental as it builds on existing topic modeling and neural language model techniques.
The authors tackled the problem of language models lacking broader document context by introducing a neural language model that incorporates document-level topic information, resulting in improved perplexity over sentence-based models and more coherent topics compared to standard LDA.
Language models are typically applied at the sentence level, without access to the broader document context. We present a neural language model that incorporates document context in the form of a topic model-like architecture, thus providing a succinct representation of the broader document context outside of the current sentence. Experiments over a range of datasets demonstrate that our model outperforms a pure sentence-based model in terms of language model perplexity, and leads to topics that are potentially more coherent than those produced by a standard LDA topic model. Our model also has the ability to generate related sentences for a topic, providing another way to interpret topics.