Modeling Local Dependence in Natural Language with Multi-channel Recurrent Neural Networks
This addresses the challenge of leveraging unparameterized semantic structures in NLP, offering a novel method for tasks such as translation and summarization, though it appears incremental as an improved variant of RNNs.
The paper tackles the problem of capturing local semantic structure in natural language by proposing Multi-Channel RNN (MC-RNN), which dynamically models dependence patterns through multiple channels and attention, resulting in significant improvements over top systems on tasks like machine translation, summarization, and language modeling.
Recurrent Neural Networks (RNNs) have been widely used in processing natural language tasks and achieve huge success. Traditional RNNs usually treat each token in a sentence uniformly and equally. However, this may miss the rich semantic structure information of a sentence, which is useful for understanding natural languages. Since semantic structures such as word dependence patterns are not parameterized, it is a challenge to capture and leverage structure information. In this paper, we propose an improved variant of RNN, Multi-Channel RNN (MC-RNN), to dynamically capture and leverage local semantic structure information. Concretely, MC-RNN contains multiple channels, each of which represents a local dependence pattern at a time. An attention mechanism is introduced to combine these patterns at each step, according to the semantic information. Then we parameterize structure information by adaptively selecting the most appropriate connection structures among channels. In this way, diverse local structures and dependence patterns in sentences can be well captured by MC-RNN. To verify the effectiveness of MC-RNN, we conduct extensive experiments on typical natural language processing tasks, including neural machine translation, abstractive summarization, and language modeling. Experimental results on these tasks all show significant improvements of MC-RNN over current top systems.