Evaluating Document Coherence Modelling
This work addresses the challenge of assessing discourse modeling capabilities in language models for NLP researchers, though it is incremental as it builds on existing evaluation methods.
The authors tackled the problem of evaluating pretrained language models' ability to model discourse coherence by proposing a sentence intrusion detection task and introducing the INSteD dataset with over 170,000 documents. Their experiments showed that these models perform well in-domain but suffer a substantial drop in cross-domain settings, indicating limited generalization capacity.
While pretrained language models ("LM") have driven impressive gains over morpho-syntactic and semantic tasks, their ability to model discourse and pragmatic phenomena is less clear. As a step towards a better understanding of their discourse modelling capabilities, we propose a sentence intrusion detection task. We examine the performance of a broad range of pretrained LMs on this detection task for English. Lacking a dataset for the task, we introduce INSteD, a novel intruder sentence detection dataset, containing 170,000+ documents constructed from English Wikipedia and CNN news articles. Our experiments show that pretrained LMs perform impressively in in-domain evaluation, but experience a substantial drop in the cross-domain setting, indicating limited generalisation capacity. Further results over a novel linguistic probe dataset show that there is substantial room for improvement, especially in the cross-domain setting.