Evaluating Text Coherence at Sentence and Paragraph Levels
This work addresses text coherence evaluation for natural language processing, but it is incremental as it builds on existing methods and tasks.
The paper tackled the problem of evaluating text coherence by proposing a paragraph ordering task and adapting existing sentence ordering methods to it, finding that a recurrent graph neural network-based model is optimal for coherence modeling except in extreme conditions.
In this paper, to evaluate text coherence, we propose the paragraph ordering task as well as conducting sentence ordering. We collected four distinct corpora from different domains on which we investigate the adaptation of existing sentence ordering methods to a paragraph ordering task. We also compare the learnability and robustness of existing models by artificially creating mini datasets and noisy datasets respectively and verifying the efficiency of established models under these circumstances. Furthermore, we carry out human evaluation on the rearranged passages from two competitive models and confirm that WLCS-l is a better metric performing significantly higher correlations with human rating than tau, the most prevalent metric used before. Results from these evaluations show that except for certain extreme conditions, the recurrent graph neural network-based model is an optimal choice for coherence modeling.