Learning Sentence Embeddings for Coherence Modelling and Beyond
This work addresses text coherence modeling for NLP applications, offering incremental improvements in performance and utility for writers and readers.
The paper tackles the problem of text coherence tasks by introducing a novel self-supervised sentence embedding method that predicts sentence locations in documents, achieving competitive performance with state-of-the-art results on multiple tasks and providing insights for improving writing and document analysis.
We present a novel and effective technique for performing text coherence tasks while facilitating deeper insights into the data. Despite obtaining ever-increasing task performance, modern deep-learning approaches to NLP tasks often only provide users with the final network decision and no additional understanding of the data. In this work, we show that a new type of sentence embedding learned through self-supervision can be applied effectively to text coherence tasks while serving as a window through which deeper understanding of the data can be obtained. To produce these sentence embeddings, we train a recurrent neural network to take individual sentences and predict their location in a document in the form of a distribution over locations. We demonstrate that these embeddings, combined with simple visual heuristics, can be used to achieve performance competitive with state-of-the-art on multiple text coherence tasks, outperforming more complex and specialized approaches. Additionally, we demonstrate that these embeddings can provide insights useful to writers for improving writing quality and informing document structuring, and assisting readers in summarizing and locating information.