Unsupervised Learning of Sentence Representations Using Sequence Consistency
This work addresses a fundamental challenge in natural language processing for researchers and practitioners, offering an incremental improvement over existing methods.
The paper tackles the problem of learning universal distributed sentence representations by proposing ConsSent, an unsupervised method that enforces consistency constraints on token sequences, achieving improved performance over strong baselines in transfer learning and linguistic probing tasks.
Computing universal distributed representations of sentences is a fundamental task in natural language processing. We propose ConsSent, a simple yet surprisingly powerful unsupervised method to learn such representations by enforcing consistency constraints on sequences of tokens. We consider two classes of such constraints -- sequences that form a sentence and between two sequences that form a sentence when merged. We learn sentence encoders by training them to distinguish between consistent and inconsistent examples, the latter being generated by randomly perturbing consistent examples in six different ways. Extensive evaluation on several transfer learning and linguistic probing tasks shows improved performance over strong unsupervised and supervised baselines, substantially surpassing them in several cases. Our best results are achieved by training sentence encoders in a multitask setting and by an ensemble of encoders trained on the individual tasks.