Improving Sequence Modeling Ability of Recurrent Neural Networks via Sememes
This work addresses a fundamental problem in natural language processing by enhancing RNNs for all downstream tasks, though it is incremental as it builds on existing sememe and RNN methods.
The paper tackled improving the sequence modeling ability of recurrent neural networks (RNNs) by incorporating sememes, resulting in evident and consistent improvements on benchmark datasets like PTB and WikiText-2 for language modeling, SNLI for natural language inference, and others for sentiment analysis and paraphrase detection, with the models also showing higher robustness against adversarial attacks.
Sememes, the minimum semantic units of human languages, have been successfully utilized in various natural language processing applications. However, most existing studies exploit sememes in specific tasks and few efforts are made to utilize sememes more fundamentally. In this paper, we propose to incorporate sememes into recurrent neural networks (RNNs) to improve their sequence modeling ability, which is beneficial to all kinds of downstream tasks. We design three different sememe incorporation methods and employ them in typical RNNs including LSTM, GRU and their bidirectional variants. In evaluation, we use several benchmark datasets involving PTB and WikiText-2 for language modeling, SNLI for natural language inference and another two datasets for sentiment analysis and paraphrase detection. Experimental results show evident and consistent improvement of our sememe-incorporated models compared with vanilla RNNs, which proves the effectiveness of our sememe incorporation methods. Moreover, we find the sememe-incorporated models have higher robustness and outperform adversarial training in defending adversarial attack. All the code and data of this work can be obtained at https://github.com/thunlp/SememeRNN.