CoUDA: Coherence Evaluation via Unified Data Augmentation
This work addresses the problem of coherence evaluation for natural language processing, offering a novel augmentation approach that improves performance over existing methods, though it is incremental in advancing data augmentation techniques.
The paper tackles the challenge of evaluating discourse coherence by proposing CoUDA, a data augmentation framework that breaks coherence into global and local aspects and designs strategies for both, achieving state-of-the-art performance with only 233M parameters, surpassing GPT-3.5 and GPT-4 based metrics in scoring and ranking tasks.
Coherence evaluation aims to assess the organization and structure of a discourse, which remains challenging even in the era of large language models. Due to the scarcity of annotated data, data augmentation is commonly used for training coherence evaluation models. However, previous augmentations for this task primarily rely on heuristic rules, lacking designing criteria as guidance. In this paper, we take inspiration from linguistic theory of discourse structure, and propose a data augmentation framework named CoUDA. CoUDA breaks down discourse coherence into global and local aspects, and designs augmentation strategies for both aspects, respectively. Especially for local coherence, we propose a novel generative strategy for constructing augmentation samples, which involves post-pretraining a generative model and applying two controlling mechanisms to control the difficulty of generated samples. During inference, CoUDA also jointly evaluates both global and local aspects to comprehensively assess the overall coherence of a discourse. Extensive experiments in coherence evaluation show that, with only 233M parameters, CoUDA achieves state-of-the-art performance in both pointwise scoring and pairwise ranking tasks, even surpassing recent GPT-3.5 and GPT-4 based metrics.