Neural Language Modeling by Jointly Learning Syntax and Lexicon
This addresses the challenge of efficiently using syntactic information in language models without requiring human-annotated data, which is incremental as it builds on existing neural and tree-structured approaches.
The paper tackles the problem of neural language modeling by jointly learning syntax and lexicon without supervision, achieving state-of-the-art performance on word/character-level language modeling tasks.
We propose a neural language model capable of unsupervised syntactic structure induction. The model leverages the structure information to form better semantic representations and better language modeling. Standard recurrent neural networks are limited by their structure and fail to efficiently use syntactic information. On the other hand, tree-structured recursive networks usually require additional structural supervision at the cost of human expert annotation. In this paper, We propose a novel neural language model, called the Parsing-Reading-Predict Networks (PRPN), that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model. In our model, the gradient can be directly back-propagated from the language model loss into the neural parsing network. Experiments show that the proposed model can discover the underlying syntactic structure and achieve state-of-the-art performance on word/character-level language model tasks.