A Hierarchical Approach to Neural Context-Aware Modeling
This work addresses the limitation of context-agnostic models in machine learning for semantic error detection, offering incremental improvements through a novel hierarchical approach.
The paper tackles the problem of enhancing neural language processing systems for semantic error detection by introducing a hierarchical recurrent neural network topology that captures context at multiple levels, resulting in relative improvements of 12.75% for unsupervised models and 20.37% for supervised models over baselines.
We present a new recurrent neural network topology to enhance state-of-the-art machine learning systems by incorporating a broader context. Our approach overcomes recent limitations with extended narratives through a multi-layered computational approach to generate an abstract context representation. Therefore, the developed system captures the narrative on word-level, sentence-level, and context-level. Through the hierarchical set-up, our proposed model summarizes the most salient information on each level and creates an abstract representation of the extended context. We subsequently use this representation to enhance neural language processing systems on the task of semantic error detection. To show the potential of the newly introduced topology, we compare the approach against a context-agnostic set-up including a standard neural language model and a supervised binary classification network. The performance measures on the error detection task show the advantage of the hierarchical context-aware topologies, improving the baseline by 12.75% relative for unsupervised models and 20.37% relative for supervised models.