Recurrent Neural Networks with Mixed Hierarchical Structures for Natural Language Processing
This work addresses the problem of improving document classification accuracy for NLP researchers and practitioners, presenting an incremental advancement in hierarchical RNN architectures.
The paper tackled the challenge of designing RNNs to learn hierarchical representations for natural language processing by proposing a multi-layer hierarchical structure with static and dynamic boundaries, focusing on document classification tasks. The resulting MHS-RNN model, enhanced with attention mechanisms, outperformed previous methods on all five tested datasets.
Hierarchical structures exist in both linguistics and Natural Language Processing (NLP) tasks. How to design RNNs to learn hierarchical representations of natural languages remains a long-standing challenge. In this paper, we define two different types of boundaries referred to as static and dynamic boundaries, respectively, and then use them to construct a multi-layer hierarchical structure for document classification tasks. In particular, we focus on a three-layer hierarchical structure with static word- and sentence- layers and a dynamic phrase-layer. LSTM cells and two boundary detectors are used to implement the proposed structure, and the resulting network is called the {\em Recurrent Neural Network with Mixed Hierarchical Structures} (MHS-RNN). We further add three layers of attention mechanisms to the MHS-RNN model. Incorporating attention mechanisms allows our model to use more important content to construct document representation and enhance its performance on document classification tasks. Experiments on five different datasets show that the proposed architecture outperforms previous methods on all the five tasks.