Semi-supervised Sequence Learning
This addresses the challenge of training recurrent networks effectively for sequence tasks, offering a semi-supervised approach that enhances stability and generalization, though it is incremental as it builds on existing pretraining and LSTM methods.
The paper tackles the problem of improving sequence learning with recurrent networks by using unlabeled data for pretraining, resulting in more stable training and better generalization that enables training on sequences up to a few hundred timesteps and strong performance in text classification tasks like IMDB, DBpedia, and 20 Newsgroups.
We present two approaches that use unlabeled data to improve sequence learning with recurrent networks. The first approach is to predict what comes next in a sequence, which is a conventional language model in natural language processing. The second approach is to use a sequence autoencoder, which reads the input sequence into a vector and predicts the input sequence again. These two algorithms can be used as a "pretraining" step for a later supervised sequence learning algorithm. In other words, the parameters obtained from the unsupervised step can be used as a starting point for other supervised training models. In our experiments, we find that long short term memory recurrent networks after being pretrained with the two approaches are more stable and generalize better. With pretraining, we are able to train long short term memory recurrent networks up to a few hundred timesteps, thereby achieving strong performance in many text classification tasks, such as IMDB, DBpedia and 20 Newsgroups.