Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations
This work addresses the need for better discourse-aware sentence embeddings in NLP, though it is incremental as it builds on existing pretrained models.
The authors tackled the problem of evaluating and improving sentence representations for discourse awareness by introducing DiscoEval, a test suite, and training objectives using Wikipedia annotations. They showed that their objectives help encode document structure aspects, with BERT and ELMo performing strongly on DiscoEval.
Prior work on pretrained sentence embeddings and benchmarks focus on the capabilities of stand-alone sentences. We propose DiscoEval, a test suite of tasks to evaluate whether sentence representations include broader context information. We also propose a variety of training objectives that makes use of natural annotations from Wikipedia to build sentence encoders capable of modeling discourse. We benchmark sentence encoders pretrained with our proposed training objectives, as well as other popular pretrained sentence encoders on DiscoEval and other sentence evaluation tasks. Empirically, we show that these training objectives help to encode different aspects of information in document structures. Moreover, BERT and ELMo demonstrate strong performances over DiscoEval with individual hidden layers showing different characteristics.