PaLM: A Hybrid Parser and Language Model
This work addresses language modeling for NLP applications, but it is incremental as it builds on existing RNN language models.
The authors tackled the problem of language modeling by introducing PaLM, a hybrid parser and neural language model that adds an attention layer over text spans in the left context, and empirically showed that it outperforms strong baselines, with further improvements when syntactic annotations are available.
We present PaLM, a hybrid parser and neural language model. Building on an RNN language model, PaLM adds an attention layer over text spans in the left context. An unsupervised constituency parser can be derived from its attention weights, using a greedy decoding algorithm. We evaluate PaLM on language modeling, and empirically show that it outperforms strong baselines. If syntactic annotations are available, the attention component can be trained in a supervised manner, providing syntactically-informed representations of the context, and further improving language modeling performance.