Multi-view Sentence Representation Learning
This work addresses the challenge of self-supervised sentence representation learning for NLP applications, but it is incremental as it builds on existing multi-view and distributional hypothesis ideas.
The paper tackles the problem of learning sentence representations by proposing a multi-view framework that combines an RNN and a linear model, trained to maximize agreement based on adjacent context information. The result shows improved representations over single-view training, with further gains from combining views and solid transferability on downstream tasks.
Multi-view learning can provide self-supervision when different views are available of the same data. The distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora. Motivated by the asymmetry in the two hemispheres of the human brain as well as the observation that different learning architectures tend to emphasise different aspects of sentence meaning, we create a unified multi-view sentence representation learning framework, in which, one view encodes the input sentence with a Recurrent Neural Network (RNN), and the other view encodes it with a simple linear model, and the training objective is to maximise the agreement specified by the adjacent context information between two views. We show that, after training, the vectors produced from our multi-view training provide improved representations over the single-view training, and the combination of different views gives further representational improvement and demonstrates solid transferability on standard downstream tasks.