Leap-LSTM: Enhancing Long Short-Term Memory for Text Categorization
This addresses the problem of processing long texts more efficiently for NLP tasks like text categorization, though it is incremental as it builds on existing LSTM and skip mechanisms.
The authors tackled the inefficiency of RNNs in processing long texts for categorization by proposing Leap-LSTM, which dynamically skips irrelevant words, resulting in faster reading and better prediction than standard LSTM and improved trade-offs between performance and efficiency compared to previous skip models.
Recurrent Neural Networks (RNNs) are widely used in the field of natural language processing (NLP), ranging from text categorization to question answering and machine translation. However, RNNs generally read the whole text from beginning to end or vice versa sometimes, which makes it inefficient to process long texts. When reading a long document for a categorization task, such as topic categorization, large quantities of words are irrelevant and can be skipped. To this end, we propose Leap-LSTM, an LSTM-enhanced model which dynamically leaps between words while reading texts. At each step, we utilize several feature encoders to extract messages from preceding texts, following texts and the current word, and then determine whether to skip the current word. We evaluate Leap-LSTM on several text categorization tasks: sentiment analysis, news categorization, ontology classification and topic classification, with five benchmark data sets. The experimental results show that our model reads faster and predicts better than standard LSTM. Compared to previous models which can also skip words, our model achieves better trade-offs between performance and efficiency.