Discourse Probing of Pretrained Language Models
This work addresses the problem of evaluating discourse understanding in pretrained language models for NLP researchers, but it is incremental as it extends existing probing methods from sentence-level to document-level tasks.
The paper introduced document-level discourse probing to evaluate pretrained language models' ability to capture document-level relations, finding BART as the best overall model but only in its encoder, with BERT performing well as a baseline, and noting substantial differences in layer effectiveness and model disparities.
Existing work on probing of pretrained language models (LMs) has predominantly focused on sentence-level syntactic tasks. In this paper, we introduce document-level discourse probing to evaluate the ability of pretrained LMs to capture document-level relations. We experiment with 7 pretrained LMs, 4 languages, and 7 discourse probing tasks, and find BART to be overall the best model at capturing discourse -- but only in its encoder, with BERT performing surprisingly well as the baseline model. Across the different models, there are substantial differences in which layers best capture discourse information, and large disparities between models.